The five-nines architecture, assembled
What you'll learn
An integration lesson, like 2.9: the complete 99.999% exchange in one diagram, walked wire-to-wire, with every functional requirement placed and every failure mode's blast radius named. No new mechanisms — this is where Part 3's five ideas become one building.
Building on
Read this as a diff against 2.9's diagram, exactly as 2.9 was a diff against 1.1. Three structural changes since four nines: the whole stack is multiplied into cells (3.4), the journal quorum now gates outputs (3.2), and a second region waits behind an async shipper (3.3) — with differential monitors (3.5) and the change pipeline (3.6) wrapped around everything.
flowchart TB
U["10M users · brokers · market makers"]
U --> RTE["Static symbol → cell routing map — versioned, cached at every gateway (3.4)"]
subgraph CELL["CELL 1 of N — a complete exchange for symbols A–F · 3 zones (2.8) · shares nothing with other cells"]
GW["Gateway fleet — idempotent sessions (2.4), ⅓ per zone"]
SEQ["Sequencer leader + risk — quorum-elected, epoch-fenced (2.6)"]
J[("Journals ×3 zones — every OUTPUT gated on quorum ack: RPO 0 (3.2)")]
E3["Engines ×3 — deterministic replicas, state digests majority-voted (2.2, 3.5)"]
MD["Feed tier — L1 at five nines, L2/L3 at four (3.1), NAK repair (2.3)"]
GW --> SEQ --> J
J --> E3 --> MD
end
RTE --> CELL
RTE --> CN["Cells 2…N — identical stacks, independent fates (3.4)"]
MD --> DLY["Delayed tier at CDN edges — millions of viewers (1.8)"]
J -- "async ship, seconds behind (3.3)" --> DR[("REGION WEST — warm standby: full stack replaying, halt-reconcile-auction runbook")]
J --> PT["Post-trade at 99.9% — clearing, settlement, surveillance (1.10, 3.1)"]
MON["Differential monitors — peer comparison across replicas, cells, feeds (3.5)"] -.-> CELL
CHG["All change: replay → shadow → canary cell → gated ramp, auto-rollback (2.5, 3.6)"] -.-> CELL
Worked example — one order, wire to wire, with everything watching
- Route: the client's gateway resolves ACME → cell 1 from its cached map — no live lookup, no shared dependency (3.4).
- Admit: a cell-1 gateway validates the session; the order carries a client ID, so any ambiguity later is retry-safe (2.4).
- Sequence and gate: the leader assigns sequence N, streams it to three zone-separated journals, and holds all outputs until 2-of-3 acknowledge — the fill that is about to exist can no longer be erased by any single failure (3.2).
- Match: three engines process event N identically; one publishes, two shadows compute digests that must agree by majority (2.2, 3.5).
- Tell everyone: the private fill returns via the session; the public print fans out through L1 (five-nines path) and depth tiers; a slow phone gets conflated truth (1.8).
- Make it real: clearing consumes event N from the journal minutes later; reconciliation replays it tonight; surveillance scores it tomorrow if need be (1.10, 3.1).
- Meanwhile: the async shipper copies event N westward within seconds (3.3), and if a deploy is in flight anywhere, it is in some other cell, behind a gate (3.6).
The consolidation table for this tier is a blast-radius table — at five nines, the design question is no longer "does it fail over?" (everything does) but "how much of the market notices, and who acts?":
| Event | Users see | Budget charged | Who acts |
|---|---|---|---|
| Process or machine dies | Nothing, or sub-second pause in one cell | ~0 | Machines (2.2, 2.6) |
| Zone lost | Seconds, one region's worth of reconnects | ~1% of the year's budget | Machines; humans watch (2.8) |
| Poison input / lockstep crash | One cell's symbols halt (1.6 semantics); rest of market trades on | 1/N of a real outage | Machines halt; humans fix (3.4) |
| Gray failure (sick, not dead) | Nothing, if the differential layer wins the race | ~0 or a per-cell blip | Machines demote; humans autopsy (3.5) |
| Bad deploy | Canary cell only, minutes, auto-reverted | 1/N, bounded by the gate | Machines roll back; humans postmortem (3.6) |
| Region lost | Market-wide halt, reopen by auction in 15–30 min | Once a decade, priced into the contract | Humans declare; runbook executes (3.3) |
Key ideas
- Five nines = cells of the four-nines stack + quorum-gated outputs + a rehearsed second region.
- Outside the cells live only facts, watchers, and the pipeline — never services trades depend on.
- The design table at this tier is blast radius × who acts, not up/down.