fivenines
31/38
Lesson 3.3 99.999%

Cell-based architecture: bounding the blast radius

🎯 Objectives
  • Partition each region into self-contained cells so no single software failure can touch more than a fixed fraction of users.
  • Use cells as the unit of deployment, capacity, and chaos.
🔗 Connect

From 2.3/2.5: shards bound data and matcher failures by city. From 2.8: canaries bound deploy failures in time. Cells generalize both: bound any failure — including the ones you haven't imagined — by construction.

A cell is a complete, self-contained copy of the T0/T1 stack — gateway, realtime, dispatch, trip, pricing, location, their data shards — serving a fixed set of cities, sharing nothing with sibling cells except the thin routing layer above and asynchronous planes below. Region A might run 8 cells of ~30 cities each. The property this buys is unlike everything before it: shards and breakers defend against failures you predicted; cells cap the damage of failures you didn't — the poison request that crashes every instance it touches, the leaked connection that exhausts a pool, the emergent retry resonance. Whatever it is, it stops at the cell wall: at 8 cells, worst case is 12.5% of a region, and because cities map to cells, it's a known list of cities — ops can name affected markets while engineers are still reading stack traces.

Cells become the unit of everything operational: deployment — the 2.8 pipeline goes cell-by-cell (canary cell first, in the quietest timezone), so even a gate-passing bad deploy hits one cell before the next gate; capacity — cells are sized identically and added like machines, turning regional scaling into stamping (1.11's infrastructure-as-code, again); chaos — 3.7 will kill cells on purpose; re-homing — 3.2 moves cities between cells across regions with the same lease machinery.

flowchart TB
  RT["Thin cell router — city to cell map, deliberately dumb"]
  RT --> C1
  RT --> C2
  RT --> C3
  subgraph C1 ["Cell A1 — 30 cities"]
    S1["Full T0/T1 stack"]
    D1[("Own shards, own caches, own dispatch")]
  end
  subgraph C2 ["Cell A2 — 30 cities"]
    S2["Full stack"]
    D2[("Own state")]
  end
  subgraph C3 ["Cell A3 — canary cell, quiet cities"]
    S3["Deploys land here first"]
    D3[("Own state")]
  end
  C1 -.-> K[["Shared async planes only — event log, observability, lakehouse"]]
  C2 -.-> K
  C3 -.-> K
Cells inside one region. The router above them holds almost no logic — anything smart enough to fail interestingly lives inside a cell, where its failure is bounded. Note the canary cell: 2.8's 1% is now a physical place.
⚠️ Pitfall

Cells rot toward sharing: someone adds a "small" shared cache, a cross-cell call for a feature, a common config endpoint — and the blast-radius guarantee silently dies. Enforce isolation mechanically (no network route between cells, separate credentials) and test it in gamedays (3.7): kill a cell, verify its siblings' SLIs don't so much as twitch.

Next step

See what actually stuck.

Take the practice scenarios now.