Measuring without lying
The measurement discipline that must precede optimization — percentiles over averages, hardware timestamps over software ones, and the classic self-deception (coordinated omission) that makes bad systems look good.
4.1's wire-to-wire boundary. You cannot climb a ladder you cannot see; this lesson builds the eyes.
Three rules, each closing a specific lie:
Rule 1 — never report averages. Latency distributions are violently skewed: a system with p50 = 20 µs and p99.9 = 8 ms has a lovely average and a terrible product, because 4.1 taught that the tail is what gets priced. Keep full histograms per hop; report p50 / p99 / p99.9 / max, and treat max as a bug report with a timestamp attached.
Rule 2 — timestamp in hardware, off the path.
flowchart TB
IN["order traffic"] --> TAP1["Passive optical tap at ingress — NIC stamps t1 in hardware"]
TAP1 --> GW["gateway"] --> SEQ["sequencer + journal quorum"] --> ME["engine"] --> PUB["publisher"]
PUB --> TAP2["Passive tap at egress — hardware t2"]
TAP1 -.copies.-> AGG["Off-path analysis: t2 − t1 per order, per hop, full histograms"]
TAP2 -.copies.-> AGG
SYNC["PTP clock sync — all taps within tens of nanoseconds"] -.-> TAP1
SYNC -.-> TAP2
Rule 3 — beware coordinated omission. The subtlest lie, and worth a worked example because most load tests commit it:
A test harness sends an order, waits for the response, then sends the next — a natural loop. The system runs at 1,000 orders/sec, then stalls completely for 10 seconds, then resumes.
- What the harness records: one slow sample (~10 s) and thousands of fast ones — because during the stall it politely stopped sending. The stall looks like one outlier: p99.9 barely moves.
- What the market would have experienced: ~10,000 orders would have arrived during those 10 seconds at the intended rate; the earliest of them would have waited nearly the full 10 seconds. The true distribution has ten thousand terrible samples, not one.
- The fix: send on a fixed schedule regardless of responses (open-loop load), and measure each order from its intended send time. The harness must model the world, and the world does not wait for your system to feel better.
Finally, wire the measurements into the machinery you already own: per-hop latency histograms are differential signals (3.5) — one gateway or one journal replica going tail-heavy relative to its peers is a gray-failure alarm long before any health check fails. And the latency SLO joins the error budget (3.6): a deploy that moves p99.9 by 20% should trip the canary gate exactly as an error-rate spike would. Measurement isn't a dashboard; it's an input to the control loops.
- Histograms and tails, never averages; max is a bug report.
- Hardware timestamps, passive taps, PTP — the instrument must outresolve the subject.
- Open-loop load or the test lies; feed latency into the 3.5/3.6 control loops.