fivenines
30/38
Lesson 3.2 99.999%

Active–active multi-region with regional homes

🎯 Objectives
  • Run T0 in three regions simultaneously, with each city homed to one region at a time.
  • Make region loss a re-homing event measured in seconds, not a failover measured in minutes.
🔗 Connect

From 2.7: active–passive and its 12 minutes. From 2.3/2.5: everything hot is already sharded by city — the crucial enabler. From 0.3: a city's riders and drivers are in the same place; their traffic can be served from the same region without conflict.

The insight that makes active–active tractable for a ride exchange: the marketplace is city-local, so we never need multi-master writes for the same data. Every city has a home region (normally the nearest); all of that city's T0 writes — pings, offers, trip transitions — go to the home region, which replicates to the others. Three regions each actively serve a third of the world, and each continuously replicates all cities' state to the other two. There is no "standby": every region is proven by production traffic every second (the 2.7 pitfall — the unproven standby — is structurally impossible).

When region A dies, its cities re-home: the global routing layer (anycast edge + a replicated, cached city→region map) flips affected cities to their designated secondary; clients reconnect (2.2's jittered reconnect); dispatch shards in the new home claim leases and rebuild markets from pings within seconds (2.5 — location's ephemerality is now the star of the whole architecture); trips resume from replicated state. Target: under 30 seconds of degraded service for affected cities, zero for everyone else — because two-thirds of the world never noticed. Compare 2.7: the same event cost 12 global minutes.

flowchart TB
  EDGE["Global edge — anycast, TLS, city-to-home routing map"]
  EDGE ==> RA
  EDGE ==> RB
  EDGE ==> RC
  subgraph RA ["Region A — home to Americas cities"]
    A0["Full T0 stack + dispatch shards"]
    AD[("City shards — primary for homed cities, replica for rest")]
  end
  subgraph RB ["Region B — home to Europe/Africa cities"]
    B0["Full T0 stack"]
    BD[("City shards — primary + replicas")]
  end
  subgraph RC ["Region C — home to Asia-Pacific cities"]
    C0["Full T0 stack"]
    CD[("City shards — primary + replicas")]
  end
  AD <-. "continuous cross-replication" .-> BD
  BD <-. " " .-> CD
  AD <-. " " .-> CD
Active–active with regional homes. One writer per city at any moment — all the availability of multi-master, none of its write conflicts. Sharding by city (2.3) was the down payment on this diagram.

Worked example: region A dies during the Americas evening peak

sequenceDiagram
    autonumber
    participant RA as Region A (dying)
    participant EDGE as Global edge
    participant HM as Home-map control plane
    participant RB as Region B (secondary for A's cities)
    participant D as Drivers in São Paulo
    participant R as Riders in São Paulo
    RA--xEDGE: health probes fail from 3 vantage points
    EDGE->>EDGE: stop routing to A (anycast withdraws) — seconds
    HM->>HM: re-home A's cities to designated secondaries (pre-computed)
    EDGE->>RB: São Paulo traffic arrives
    D->>RB: apps reconnect with jittered backoff
    Note over RB: dispatch shards claim São Paulo leases, market rebuilds from next pings (≤ 4 s)
    R->>RB: in-flight trip state? served from replica — promoted to primary
    RB-->>R: trip resumes, live map repopulates
    Note over EDGE,RB: affected cities: ~20-30 s degraded. Rest of world: zero impact. Humans: paged to watch, not to act.
Region loss as a routing event. Every step is a mechanism you already own — re-homing is 2.5's lease claim plus 2.7's promotion, running at the speed of routing instead of the speed of runbooks.
⚠️ Pitfall

The re-homing decision itself must not require the dying region (no control plane singly homed anywhere) and must resist flapping — a region that dies and returns every 90 seconds must not drag cities back and forth. Re-home fast, return slowly and deliberately: asymmetric damping, same instinct as the autoscaler in 2.2.

Next step

See what actually stuck.

Take the practice scenarios now.