Jitter: the tail is the product
Where the rare, ruinous outliers come from once the median is fast — and the discipline of hunting variance instead of speed.
4.1 argued variance is priced at its worst case; 4.2 built the histograms that expose it. This lesson is the hunt. It is also 3.5's gray-failure thinking turned inward: a machine that is fast 99.9% of the time is sick 0.1% of the time, and that sickness is the product defect.
A hot path tuned per 4.4 will show a beautiful p50 — and, untreated, a p99.9 dominated by events that have nothing to do with your algorithm:
flowchart TB
GC["Garbage collection / allocator pauses — ms-scale stalls"] -.-> HP["THE HOT PATH CORE — 2 µs median, minding its own business"]
PF["Page faults — first touch, swap, page cache misses"] -.-> HP
IRQ["Interrupts routed to the wrong core"] -.-> HP
SCH["OS housekeeping — timers, scheduler ticks, kernel threads"] -.-> HP
PWR["CPU frequency and sleep-state transitions"] -.-> HP
NN["Noisy neighbors — other processes sharing L3 cache and memory bandwidth"] -.-> HP
HP --> R["Result: p50 = 2 µs, p99.9 = 4 ms — a thousand-to-one tail, priced into every spread (4.1)"]
The countermeasure per beast: no GC on the hot path (4.4's pools; or a language without one); pre-touch and lock all memory at startup so page faults happen before the market opens; route interrupts and OS housekeeping to designated junk cores, leaving hot cores isolated from the scheduler entirely; fix CPU frequency and disable deep sleep states (a core waking from sleep costs tens of µs — steady moderate speed beats bursty top speed); and give hot processes dedicated sockets or cache partitions so batch jobs can't flush their L3. Then warm up: replay yesterday's traffic through the engine before the open (1.7's log, useful again) so code paths are compiled, branch predictors trained, and caches populated before the first real order.
An engine in a garbage-collected runtime: median 8 µs, but a 20 ms collection every 60 seconds.
- Fraction of time stalled: 20 ms / 60 s ≈ 0.033% — so roughly 1 order in 3,000 eats up to 20 ms. Your p99.9 is ruined and your max is 2,500× your median, every minute of every day.
- Who it hits: 2.7 taught that stress concentrates events — the pause is most likely to coincide with a burst, when 20 ms of queued orders pile up behind it (and 4.2's coordinated-omission lens says the queue's wait counts too).
- What the market does: market makers observe the outliers (they measure you better than you measure yourself — 4.2's taps work from outside too) and widen spreads to price a 20 ms adverse-selection window. Everyone pays for your allocator, forever.
Two systemic notes. First, jitter compounds exactly like the serial hops of 4.3 — every stage's rare stall sums into the chain's tail, which is why the discipline must hold at every hop, gateway to publisher. Second, close the loop with the differential machinery: per-hop, per-instance p99.9 compared across peers (3.5) turns "one machine started jittering" — a failing fan, a BIOS update that re-enabled sleep states, a neighbor process — into an alarm and a demotion before market makers price it. Reliability engineering and latency engineering, one control loop again.
- Once the median is fast, the platform — not the algorithm — owns your tail.
- Isolate, pre-touch, pin, fix frequencies, warm up; trade median for tail every time.
- Feed p99.9 into the 3.5 differential loop — jitter is gray failure with a price tag.