The microsecond rung: mechanical sympathy
What must change when the enemy is no longer your architecture but your operating system and your memory hierarchy — kernel bypass, pinned cores, and data structures shaped for the cache.
Architecture-level budgets removed unnecessary services and encodings. Below a millisecond, operating-system scheduling and memory hierarchy dominate the remaining variance.
A natural first design is to keep the ordinary socket, interrupt, allocator, and scheduler path and search for smaller savings inside the matching algorithm. But the hot leg has one pinned ordered execution path with preallocated cache-friendly state, explicit NUMA placement, and no involuntary kernel or allocation work in the steady state.
The kernel is a general-purpose compromise, and the hot path can't afford compromises. Every conventional network receive crosses it: interrupt, kernel-buffer copy, protocol stack, syscall wakeup, scheduler decision, context switch — tens of microseconds each way, with variance from everything else the machine is doing. The sub-millisecond rung removes the kernel from the data plane:
flowchart TB
subgraph K["Kernel path — fine for the account APIs"]
K1["NIC raises interrupt"] --> K2["kernel copies into buffers, runs stack"] --> K3["syscall boundary"] --> K4["scheduler wakes app thread — eventually"]
end
subgraph B["Bypass path — the trading hot path"]
B1["NIC writes packet directly into application memory"] --> B2["dedicated core, pinned, busy-polling — never sleeps, never scheduled"] --> B3["message read in place: zero copies, zero syscalls, zero context switches"]
end
K4 -."tens of µs, variable".-> OUT1["order enters app"]
B3 -."single-digit µs, steady".-> OUT2["order enters engine loop"]
Then aim the same sympathy at memory. A cache miss to main memory costs ~100 ns — at this rung, a budget item. Three consequences for the engine 1.3 designed:
- The book is arrays, not trees. Textbooks suggest balanced trees for sorted price levels; trees chase pointers, and every hop is a potential cache miss. Real engines exploit domain facts: prices move on a tick grid (1.5) and activity clusters violently near the top of book (1.3) — so a contiguous array of price levels indexed by tick, plus intrusive FIFO queues per level, makes the hot operations (match at best, add near best) touch one or two warm cache lines. The data structure is chosen by the memory hierarchy, not by big-O alone.
- No allocation on the hot path. Every order, level, and message object is pre-allocated in pools at startup. Allocation cost isn't the malloc — it's the page faults, the cache pollution, and (in managed languages) the GC pauses that 4.5 will show ruin the tail.
- Single-threading pays again. The 1.3 decision — one core owns the book — now delivers its third dividend: no locks, no cache lines ping-ponging between cores, and a branch predictor that learns the matching loop cold. The availability design (1.7 determinism) and the latency design keep turning out to be the same design.
Budget at this rung: NIC-to-engine ~2 µs (bypass), risk + match ~1–3 µs (warm cache, no allocation), engine-to-wire ~2 µs, journal quorum on metro-close zones ~200–400 µs — wire-to-wire p99 ≈ 500 µs, and now one line item is 80% of the budget. Stare at that: the quorum. The entire remaining ladder is a negotiation with it, and 4.7 conducts the negotiation.
A polling thread runs on one NUMA node while the book's memory is allocated on another, or an interrupt migrates it between cores. The algorithm is unchanged, yet cache misses and scheduler noise dominate the tail.
Below ~500 µs, evict the kernel from the data plane: bypass NICs, pinned busy-polling cores, ring buffers. Shape data for the cache: tick-indexed books, pre-allocated pools, zero hot-path allocation. Single-threaded determinism pays its third dividend; the quorum becomes 80% of the budget.