fivenines
32/39
Lesson 3.8

Capstone: the whole exchange, end to end

What you'll learn

Nothing new — deliberately. This lesson assembles every mechanism into one picture and one table, then hands you the schema to keep.

flowchart TB
    U["10M users · 500k sessions · brokers · market makers"]
    U --> RTE["Static symbol → cell routing (3.4)"]
    subgraph CELL["ONE CELL of N — full stack, shares nothing (3.4) — REGION EAST, 3 zones (2.8)"]
        GWF["Gateway fleet — stateless, N+2, idempotent sessions (1.1, 2.4)"]
        SEQC["Sequencer + risk — quorum leader, epoch-fenced (1.9, 2.6)"]
        JQ[("Journal ×3 zones — quorum-gated outputs, RPO 0 (1.7, 3.2)")]
        ENG["Engines ×3 — deterministic replicas, digests compared (1.3–1.6, 2.2, 3.5)"]
        MDF["Feed tier — L1/L2/L3, conflation, NAK repair (1.8, 2.3)"]
        GWF --> SEQC --> JQ
        JQ --> ENG --> MDF
    end
    RTE --> CELL
    MDF --> EDGE["Delayed tier at CDN edges — millions of viewers (1.8)"]
    JQ -- async ship (3.3) --> DR[("REGION WEST — warm standby, halt-reconcile-auction runbook")]
    JQ --> PTF["Post-trade: clearing, settlement, surveillance, audit — async consumers at 99.9% (1.10, 3.1)"]
    CHG["All change: replay → shadow → cell canary → gated ramp (2.5, 3.6)"] -.-> CELL
    
The five-nines exchange. Every box cites the lessons that built it. Trace any failure — process, zone, region, poison input, overload, gray sickness, bad deploy — and the diagram names the mechanism that catches it and the lesson that explains why.

The ladder you climbed, as one table — read a row per part, or a column per idea:

99.9% — Part 199.99% — Part 299.999% — Part 3
StateOne deterministic copy; journal + snapshotsLive replicas consuming the log; quorum journalsQuorum-gated outputs (RPO 0); state digests voted
FailureRestart and replay, minutesAutomated failover, seconds; fencing kills split-brainRegion loss survivable; gray failure hunted differentially
ChangeMaintenance windowsRolling edge, failover upgrades, replay-diff, N/N−1Cell-by-cell ramps, automated burn-rate rollback
HumansDecide during incidentsWatch; machines decideRehearse; drills make real incidents reruns
Blast radiusThe shardThe zoneThe cell — 1/N of the market, by construction

And the schema beneath the schema — five ideas that did nearly all the work, worth carrying to any system you build next:

  1. Determinism is the master key. State = f(log) bought recovery, replication, audit, testing, shadow deploys, and divergence detection — one property, seven products (1.7 → everywhere).
  2. Put things off the critical path. Every component moved off the hot path (1.1, 1.10, 3.4's aggregator) stopped costing availability there. The cheapest nine is the one a component no longer needs.
  3. Quorums turn opinion into arithmetic. Who leads (2.6), what's durable (3.2), who's lying (3.5) — all the same trick: majorities of a fixed group can't disagree.
  4. Make the failure path the practiced path. Untested standbys are rumors (2.1); drilled ones are five nines (3.5). Reliability lives in the paths you exercise, not the ones you diagram.
  5. Each nine changes who acts. Three nines: humans repair. Four: machines repair, humans watch. Five: machines repair, humans rehearse — and the remaining enemy is your own change (3.6).

Where to go deeper: the replicated-state-machine and consensus literature (Lamport's Paxos, Ongaro & Ousterhout's Raft, viewstamped replication); the LMAX architecture write-ups for deterministic matching at speed; Aeron and its cluster for sequenced messaging in practice; the Google SRE book for error budgets and burn-rate alerting; and the "Gray Failure" paper (Huang et al., 2017) for the failure mode Part 3 hunts. Every one of them will now read as elaboration of diagrams you already hold.

The course in one sentence

Build a small deterministic core fed by one sequenced log, keep everything else off its path, replicate it with quorums, cap every blast radius, rehearse every failure — and each nine becomes a budget you allocate, not a miracle you hope for.

Next step

See what actually stuck.

Take the practice scenarios now.