Lesson 2.12 99.99%
Recap: the four-nines checklist
🔗 Connect
Consolidation again. Part 2 in one claim: no single failure — instance, zone, shard, dependency, deploy, or region — requires a human to keep the exchange inside its SLO. Verify it against the 1.14 table.
| 1.14 failure row | Part 2 answer | Mechanism |
|---|---|---|
| Database primary dies | ~20 s automated failover, zero loss | Quorum orchestrator + fencing + client retries (2.3) |
| Bad deploy | 1% blast radius, auto-rollback in minutes | Canary + metrics gate + flags (2.8) |
| Vendor brownout | Feature degrades, flow survives | Timeouts, breakers, bulkheads, fallbacks (2.6) |
| Traffic spike | T0/T1 goodput preserved | Tiered shedding + N+1 floors (2.6, 2.2) |
| Cascade via sync calls | Fan-out decoupled, consumers replay | Outbox + event log + idempotency law (2.4) |
| Matcher state loss | ~15 s rebuild, invariants intact | Leases + rebuild-from-stream (2.5) |
| Region failure | ~12 min evacuation, bounded RPO | Warm standby + rehearsed runbook (2.7) |
| Detection for all of the above | Pages proportional to user harm | Flow SLIs + burn-rate alerts (2.1, 2.9) |
flowchart LR F["Any single failure"] --> DET["Detected by burn-rate SLIs — seconds to minutes"] DET --> REC["Recovered by automation — no human in loop"] REC --> CON["Contained by shards, tiers, bulkheads, canaries"] CON --> OK["SLO holds: ≤ 4 m 23 s per 30 d"] F -. "the residual" .-> GAP["Region failover still costs ~12 min — one event can exceed a year at five nines"] GAP --> P3["Part 3: stop failing over. Be already there."]