PFX Build 22: Recover in a Standby Region

Losing a datacenter gracefully

The region problem: why the metro that hosts your three zones is itself a single point of failure, and the honest trade-offs of the standby region.

Three zones protect the venue from one site loss, but the metro containing all three zones is still one correlated failure domain.

A natural first design is to automatically promote the remote region whenever it loses contact with the primary, as though inter-region latency and partitions were ordinary zone failures. But exactly one region trades; regional failover is a deliberate authority transfer that halts, declares, reconciles shipped state, and reopens by auction under a stated recovery-point bound.

Part 2's three zones share a metro area — and metros fail: grid collapses, fiber cuts, floods, regional cloud outages. Rare, but five nines is precisely the business of pricing rare. The instinct is symmetry: "active-active across regions!" Physics vetoes it. Regions sit 10–60 ms apart; putting that into the 3.2 quorum path would multiply order latency ~20×, and a market maker 40 ms from the matching engine is not attending the same market as one 0.5 ms away — you'd have built two half-fair exchanges. For a trading core, the honest design is active region + warm standby region:

flowchart LR
    subgraph RE["REGION EAST — active"]
        CORE["Core: 3 zones, quorum journal (3.2)"]
        JE[("Journal — synchronous, RPO 0")]
    end
    subgraph RW["REGION WEST — warm standby"]
        JW[("Journal copy — async, seconds behind")]
        CW["Full core stack, replaying the stream, outputs suppressed"]
        GW2["Gateway + feed fleet, idle-hot"]
    end
    CORE --> JE
    JE -- "async ship, lag L ≈ 1–5 s" --> JW
    JW --> CW
    

Asynchronous means honest bookkeeping: if East vanishes at instant T, West holds the journal through T − L. Events inside the lag window are gone — RPO of seconds, not zero. For most systems that's a footnote; for an exchange, those seconds may contain published fills that West's journal never received. So regional failover is not the silent promotion of 2.8 — it's a deliberate, procedural event:

1. Declare (human decision, minutes — correctly so: this is the one failover where a wrong guess loses data, and the 3.1 budget prices one such event per decade, not per month).
2. Halt-and-reconcile: West opens in Halted (1.6's machinery, reused). Every session reconnects via 2.4 replay against West's journal head; orders and fills inside the lag window surface as sequence gaps — provably identified, per client, rather than silently divergent. Client IDs (2.4) make resubmission safe.
3. Reopen via auction (1.6, again): post-halt price formation aggregates everyone's post-disaster interest instead of letting the first panicked order set the price.

Target: trading resumes in West within 15–30 minutes of a region-killing event — an event with a once-a-decade probability, which the 3.1 shared budget absorbs. The regional design adds little new machinery: a journal shipper and a runbook. Halts, auctions, session replay, standby cores — all established earlier. At five nines, region survival is mostly the discipline of rehearsing a path made of parts you already trust (3.5 makes rehearsal a first-class practice).

Both regions are healthy but the inter-region link fails. Automatic promotion creates two venues accepting orders against diverging books. The slower procedural path is intentional because preventing dual trading outranks shaving seconds from a rare regional disaster.

Regions are the zone problem at the next scale — but latency physics forbids the symmetric fix. Async standby region: RPO of seconds, procedural failover, reconcile-then-auction. Region survival = old mechanisms + a shipper + a rehearsed runbook.