fivenines

Lesson 3.1 99.999%

Where five nines is worth it — and where it's waste

🎯 Objectives

Derive which flows get the fifth nine from user physics and dependency arithmetic.
Restructure T0 so nothing synchronous sits beneath it with lower availability.

🔗 Connect

From 2.1: the tier table. From 0.3: serial chains multiply; a five-nines flow cannot synchronously depend on a four-nines anything. From 2.12's residual: 12-minute failovers are 27× one month's budget.

Start from user physics, not vanity. A rider mid-trip with a frozen app is a safety incident; a driver who can't go online is lost income by the minute; these justify extreme cost. A receipt arriving an hour late is nothing. So the fifth nine goes to exactly four flows: location ingest + nearby query, dispatch offer/accept, trip state transitions, and the realtime channel that carries them. Payments stay at four nines by design: the authorize is quote-time (T1), the capture is already asynchronous (1.8, 2.4) — money's durability is absolute but its latency is forgiving. This is the sharpest lesson of the whole course: we bought payments' five-nines user experience with an async pattern instead of five-nines infrastructure. Restructuring beats gold-plating.

Then enforce the arithmetic downward: everything synchronously beneath a T0 flow must be ≥ 99.999% itself, or leave the synchronous path. Audit dispatch's remaining sync edges: geo index (stays — engineered up with it), trip store conditional assign (stays — engineered up), refined ETA from the Geo facade (leaves — dispatch falls back to straight-line-distance ranking if the facade is slow; a slightly worse match beats no match), fraud scoring (already fail-open, 2.11), pricing (already quote-time). The T0 dependency tree after the audit is five components deep in total — that's what makes the fifth nine even arithmetically possible.

flowchart TB
  subgraph T0 ["T0 — 99.999%: the exchange core"]
    ING["Location ingest + nearby query"]
    DISP["Dispatch offer / accept"]
    TS["Trip state transitions"]
    RTG["Realtime channel"]
  end
  subgraph T1 ["T1 — 99.99%"]
    Q["Quotes"]
    AUTH["Payment auth + capture"]
    LOGIN["Login"]
  end
  subgraph T2 ["T2/T3 — 99.9% and below"]
    HIST["History, ratings, receipts, chat"]
    BI["Lakehouse, BI, reporting"]
  end
  DISP -- "sync, engineered to 5 nines" --> ING
  DISP -- "sync, engineered to 5 nines" --> TS
  DISP -. "async or fallback only" .-> Q
  TS -. "events only (2.4)" .-> AUTH
  TS -. "events only" .-> HIST
  AUTH -. "events only" .-> BI

The tier lattice with edges typed. Solid = synchronous, must match tier. Dashed = asynchronous or fallback — the tier boundary crossing that's allowed. Five nines is as much about deleting solid edges as strengthening components.

Where five nines is worth it — and where it's waste

See what actually stuck.