Losing a datacenter gracefully

Part 2's three zones share a metro area — and metros fail: grid collapses, fiber cuts, floods, regional cloud outages. Rare, but five nines is precisely the business of pricing rare. The instinct is symmetry: "active-active across regions!" Physics vetoes it. Regions sit 10–60 ms apart; putting that into the 3.2 quorum path would multiply order latency ~20×, and a market maker 40 ms from the matching engine is not attending the same market as one 0.5 ms away — you'd have built two half-fair exchanges. For a trading core, the honest design is active region + warm standby region:

flowchart LR
    subgraph RE["REGION EAST — active"]
        CORE["Core: 3 zones, quorum journal (3.2)"]
        JE[("Journal — synchronous, RPO 0")]
    end
    subgraph RW["REGION WEST — warm standby"]
        JW[("Journal copy — async, seconds behind")]
        CW["Full core stack, replaying the stream, outputs suppressed"]
        GW2["Gateway + feed fleet, idle-hot"]
    end
    CORE --> JE
    JE -- "async ship, lag L ≈ 1–5 s" --> JW
    JW --> CW
    

Asynchronous means honest bookkeeping: if East vanishes at instant T, West holds the journal through T − L. Events inside the lag window are gone — RPO of seconds, not zero. For most systems that's a footnote; for an exchange, those seconds may contain published fills that West's journal never received. So regional failover is not the silent promotion of 2.8 — it's a deliberate, procedural event:

1. Declare (human decision, minutes — correctly so: this is the one failover where a wrong guess loses data, and the 3.1 budget prices one such event per decade, not per month).
2. Halt-and-reconcile: West opens in Halted (1.6's machinery, reused). Every session reconnects via 2.4 replay against West's journal head; orders and fills inside the lag window surface as sequence gaps — provably identified, per client, rather than silently divergent. Client IDs (2.4) make resubmission safe.
3. Reopen via auction (1.6, again): post-halt price formation aggregates everyone's post-disaster interest instead of letting the first panicked order set the price.

Target: trading resumes in West within 15–30 minutes of a region-killing event — an event with a once-a-decade probability, which the 3.1 shared budget absorbs. Note how little new machinery this lesson introduced: a journal shipper and a runbook. Halts, auctions, session replay, standby cores — all built lessons ago. At five nines, region survival is mostly the discipline of rehearsing a path made of parts you already trust (3.5 makes rehearsal a first-class practice).