fivenines
37/38

Theory Tutorial

The operating model and the cost of a nine

Time
5m
Level
not specified
Artifacts
theory + practice
Progress0%
Lesson 3.9 99.999%

The operating model and the cost of a nine

From 2.1: the error budget converts reliability into currency. Now run the books: what did each Part actually buy, and at what price? From 3.1: the tier lattice was the cost-control mechanism all along.

The obvious design is to treat higher availability as an unconditional engineering improvement and optimize its infrastructure cost later. It makes the happy path simple, but it protects the wrong property. A reliability target is justified only when the user and business loss it prevents exceeds its infrastructure, engineering, and delivery cost.

Part 1 — 99.9%Part 2 — 99.99%Part 3 — 99.999% (T0 only)
Infra cost (norm.)~2–2.5× — standby region, event backbone, N+1 floors, streaming platform~3.5–4.5× — three active regions, cells, 5-way quorums, probe fleet (T0 only; tiering is why it isn't 8×)
EngineeringProduct-dominant~25–30% on reliability machineryDedicated platform/SRE org; chaos and remediation are products with roadmaps
Velocity taxShip on greenCanary gates add hours; budget freezes biteCell-by-cell rollouts add days; every T0 change carries a fault-injection review
Marginal user-minutes saved~39 min/month~4 min/month

Read the last row twice: the fourth nine saved ~39 user-facing minutes a month; the fifth saves ~4 more — at roughly double the total cost. For receipts, that trade is absurd (3.1 said so; now you see the arithmetic). For the exchange core it's justified by what those minutes are: at 1,000 requests/s, 4 minutes is a quarter-million stranded matching attempts, safety exposure for riders mid-trip, and market-share bleed in a business where the other app is one thumb-swipe away. The uncomfortable, senior-engineer truth: the correct availability target is an economic conclusion, not an engineering aspiration — and the tier lattice is how you buy each flow only the nines it earns.

The operating rules that keep the system there are not optional: every T0 change rides the full progressive path, with no exception for "trivial" config (2.8); error budgets enforce freezes automatically (2.1); the fault catalog runs continuously and a failed chaos experiment blocks related launches (3.7); postmortem actions are funded before feature work when budget is spent (2.9); dependency reviews gate any new solid edge under T0 (3.1); and on-call staffing treats operator attention as a finite resource. One incident commander cannot coordinate three simultaneous compound failures; pretending otherwise is how five-nines systems have four-nines months.

flowchart LR
  N3["99.9% — 43.8 m/mo down"] -- "+~39 min saved · ~2× cost" --> N4["99.99% — 4.4 m/mo"]
  N4 -- "+~4 min saved · ~2× cost again" --> N5["99.999% — 26 s/mo"]
  N5 -.-> WHERE{"Worth it where?"}
  WHERE -- "exchange core: safety, revenue, trust" --> YES["T0 — pay it"]
  WHERE -- "everything else" --> NO["T1-T3 — buy the async pattern instead (3.1)"]
Diminishing minutes at compounding cost. The architecture's cleverest move was never a technology — it was the tier lattice deciding where not to spend.

Now consider what happens when a cost reduction removes T0 redundancy but leaves low-value reporting at the same target. If the central guarantee disappears here, the design has confused an optimization with a correctness boundary.

Next step

See what actually stuck.

Take the practice scenarios now.