Multi-region disaster recovery: active–passive
- Stand up a warm standby region with continuous replication and define RTO/RPO per data class.
- Design region evacuation as a rehearsed, one-decision procedure.
From 1.14's table: region loss is the row that exceeds a year of budget in one event. From 2.1: at four nines you get 52 minutes per year — one unrehearsed region failover could spend all of it. From 1.11: infrastructure-as-code is what makes a second region stampable at all.
Two vocabulary words govern everything here. RTO (recovery time objective): how long until service resumes. RPO (recovery point objective): how much recently-written data you may lose. We run active–passive: a warm standby region receives asynchronous replication of every datastore, runs a skeleton of every service (scaled low but proven alive by a trickle of mirrored traffic), and can take the world in under 15 minutes. Async replication means a non-zero RPO — seconds of the most recent writes may be lost in a true region-death — so we set RPO per data class deliberately: ledger and trips replicate with the tightest lag and reconcile on recovery (idempotency keys and PSP records let us reconstruct in-flight payments); location state has RPO = don't care — it rebuilds from pings in 4 seconds (1.4's ephemerality pays off again).
flowchart TB
DNS["Global DNS / traffic manager — health-checked, low TTL"]
DNS ==> RA
DNS -. "standby route" .-> RB
subgraph RA ["Region A — active"]
SA["All services — full scale"]
DA[("Shards, ledger, global DB — primaries")]
KA[["Event log"]]
end
subgraph RB ["Region B — warm standby"]
SB["All services — skeleton scale + trickle traffic"]
DB2[("Replicas — async, lag monitored per class")]
KB[["Event log — mirrored"]]
end
DA -. "async replication" .-> DB2
KA -. "mirror" .-> KB
RUN["Evacuation runbook — one decision, then automation"] --- DNS
Worked example: evacuating the region
sequenceDiagram
autonumber
participant MON as Monitoring
participant IC as Incident commander
participant AUTO as Evacuation automation
participant B as Region B
participant DNS as Traffic manager
participant APPS as Rider and driver apps
MON->>IC: region A: multi-zone power event, all SLOs red
IC->>IC: one human decision: evacuate (criteria pre-agreed, ~5 min)
IC->>AUTO: execute region failover
AUTO->>B: promote replicas (per class, honoring RPO order: ledger, trips, global)
AUTO->>B: scale services skeleton to full (pre-warmed images, floors)
AUTO->>DNS: flip active route to region B
APPS->>DNS: reconnect on failure — resolve to B
APPS->>B: sessions resume (stateless tokens — 1.2 pays off)
B->>B: dispatch shards rebuild markets from pings (2.5 pays off)
Note over B: trips in flight resume from replicated state, riders see a reconnect blip
B->>IC: serving, SLOs green — total 12 min
IC->>IC: begin reconciliation: RPO-window writes vs PSP + client caches
The standby that can't absorb the load: region B at 10% capacity meets a 100% traffic flip and immediately melts — you've traded an outage for an outage plus confusion. Pre-agree the scale-up path (quotas reserved with the cloud provider, floors that autoscale from trickle to full in minutes) and rehearse the flip quarterly with real traffic. DR that hasn't run is DR you don't have.