Design PFX: Four nines at scale: multi-zone anatomy

Lesson 2.8

Four nines at scale: multi-zone anatomy

What you'll learn

The physical layout that keeps Part 2's logical redundancy honest — and a full-dress rehearsal of a zone failure to close the part.

Every mechanism in Part 2 assumed replica failures are independent (0.3's warning). Physically co-located replicas share power, cooling, and switches — one electrical fault takes primary and standby together, and your elaborate failover machinery fences a pile of equally-dead hardware. Independence must be arranged in the physical world: availability zones — datacenter facilities close enough for sub-millisecond latency, separate enough to fail independently.

flowchart TB
    subgraph ZA["Zone A"]
        GA["Gateways ⅓"]
        PA["Engine PRIMARY"]
        JA[("Journal replica")]
        EA["Election node"]
    end
    subgraph ZB["Zone B"]
        GB["Gateways ⅓"]
        PB["Engine STANDBY"]
        JB[("Journal replica")]
        EB["Election node"]
    end
    subgraph ZC["Zone C"]
        GC["Gateways ⅓"]
        PC["Second STANDBY"]
        JC[("Journal replica")]
        EC["Election node"]
    end
    PA == "sequenced stream (2.3)" ==> PB
    PA ==> PC
    PA --> JA & JB & JC

One shard across three zones. Every replicated thing — engines, journals, election nodes, gateway capacity — spreads so that no single zone holds a majority of anything or the only copy of anything. Three zones, not two: quorums (2.6) need a majority to survive one zone's loss.

Sizing rule: every pool runs at ≤ ⅔ capacity, so losing one zone of three leaves enough to carry full load — redundancy you can't afford to use is decoration. And the market-data fan-out tree (1.8) gets one more payoff here: only its root lives in the core's zones; feed servers and delay caches push outward toward users, so the fan-out — the biggest resource consumer in the system — scales and fails independently of the trading core.

Worked example — Part 2's final exam: Zone A loses power at 14:02:00.0

+0.0 s: Primary engine, one journal replica, one election node, a third of the gateways: gone, simultaneously.
+0.5 s: Election nodes in B and C miss heartbeats; lease expires; B's candidate wins with 2-of-3 majority. Epoch increments — anything Zone A might still emit is fenced (2.6).
+1.5 s: Zone B standby confirms it has processed to the stream head — it was never behind (2.2) — and starts publishing. Journal quorum: 2-of-3 replicas still acknowledging (2.3).
+2–10 s: Zone A's clients reconnect through B and C gateways; session replay proves order state to every one of them (2.4). Remaining gateways absorb the load at their reserved headroom.
Downtime charged: a few seconds of order-entry pause. Budget spent: ~1% of the year's 53 minutes. No human decided anything; the on-call was paged to watch, not to act.

That walk-through is Part 2 in one incident: replication (2.2) over reliable streams (2.3), sessions that reattach (2.4), quorum + fencing deciding leadership (2.6), physical layout guaranteeing the survivors existed (2.8). What it does not survive: the whole metro region, a poison message crashing all three replicas in lockstep, or a bad deploy propagating everywhere at once. Those need another nine — and another part.

Key ideas

Independence is physical: three zones, majorities of nothing in any one.
Run pools at ⅔ so surviving capacity carries full load.
A zone loss should cost seconds and zero human decisions — that's four nines, achieved.

Four nines at scale: multi-zone anatomy

See what actually stuck.