Consensus in the hot path
The last durability gap in the design — outputs that outrun the journal — and how quorum acknowledgment closes it for the price of a round trip.
Lesson 2.9 found the final window: an output may become visible before its causal event survives failure. Quorum election alone does not close that window.
It is tempting to publish the fill immediately for low latency and replicate the event a few milliseconds later. The constraint it misses is simple: No output leaves the core until the event that caused it is durable on a zone-separated quorum; an event without quorum can produce no externally observed consequence.
A confession about Part 2: the primary journaled every event to a quorum (2.3), but nothing said when. If the engine publishes a fill and journal replication completes a few milliseconds later, there is a window where a perfectly-timed crash produces the unforgivable: a fill the counterparty saw, that recovery — replaying a journal that never received the event — says never happened. At four nines, that microscopic window was an accepted risk. The five-nines core closes it with a rule of one sentence: no output leaves the core before the event that caused it is acknowledged by a quorum of journal replicas in distinct zones.
sequenceDiagram
autonumber
participant G as Gateway
participant L as Leader (sequencer + engine)
participant JA as Journal · Zone A
participant JB as Journal · Zone B
participant JC as Journal · Zone C
G->>L: order, client ID X-9001
L->>JA: append event 7,340,112
L->>JB: append event 7,340,112
L->>JC: append event 7,340,112
JA-->>L: ack
JB-->>L: ack — quorum: 2 of 3, in distinct zones
L->>L: NOW process: match, generate fill
L-->>G: fill published — durably backed
Note over JC: acks late or never — quorum already sufficed
The price is latency: one zone-to-zone round trip (~0.5–2 ms) added before any ack or fill. Three ways the design keeps that from hurting:
- Pipeline, don't stall. The engine doesn't idle awaiting acks for event N — it sequences and replicates N+1, N+2, … continuously, releasing each event's outputs only as its quorum lands. Throughput is unaffected; only per-event latency pays the round trip.
- Wait for the quorum, not the slowest. 2-of-3 means the laggiest zone (GC pause, network wobble) never sits on the critical path — a quiet defense against gray failure (3.5) built into the arithmetic.
- Failover inherits the guarantee. The new leader (2.6) reads forward from the journal quorum before publishing: every quorum-acked event is provably there, every non-acked event provably produced no output, so clients' 2.4 replay is exact even across a leader death. The unforgivable window is closed by construction, not by luck.
Where does this stop? At the five-nines boundary from 3.1. The trading core pays the round trip because a lost fill is intolerable. Market data fan-out doesn't — a feed relay crash loses nothing (consumers re-request from retransmit servers, 2.3). Post-trade doesn't — it reads the now-bulletproof journal. One sentence of policy, applied exactly where its cost is justified.
The third journal is slow or lost after two acknowledgments. Waiting for all three adds a gray-failing replica to the critical path without improving one-zone survival. Two acknowledgments establish the guarantee; pipelining protects throughput while output alone waits.
Gate outputs on per-event quorum durability across zones: RPO = 0. Pipeline replication to protect throughput; quorum beats the slowest replica. Pay the round trip only inside the five-nines boundary.