fivenines
31/38

Guided Problem

YourRide Build 26: Contain a Bad Matcher Release

Time
30m
Level
advanced
Artifacts
not specified
Progress0%
Lesson 3.3 99.999%

Cell-based architecture: bounding the blast radius

From 2.3/2.5: shards bound data and matcher failures by city. From 2.8: canaries bound deploy failures in time. Cells generalize both: bound any failure — including the ones you haven't imagined — by construction.

At first glance, it seems reasonable to share caches and internal services between cells to improve utilization and reduce duplication. The choice optimizes the common case by weakening the boundary that matters. A cell owns its critical state and capacity so unknown software, dependency, and overload failures have a fixed user blast radius.

A cell is a complete, self-contained copy of the T0/T1 stack — gateway, realtime, dispatch, trip, pricing, location, their data shards — serving a fixed set of cities, sharing nothing with sibling cells except the thin routing layer above and asynchronous planes below. Region A might run 8 cells of ~30 cities each. The property this buys is unlike everything before it: shards and breakers defend against failures you predicted; cells cap the damage of failures you didn't — the poison request that crashes every instance it touches, the leaked connection that exhausts a pool, the emergent retry resonance. Whatever it is, it stops at the cell wall: at 8 cells, worst case is 12.5% of a region, and because cities map to cells, it's a known list of cities — ops can name affected markets while engineers are still reading stack traces.

Cells become the unit of everything operational: deployment — the 2.8 pipeline goes cell-by-cell (canary cell first, in the quietest timezone), so even a gate-passing bad deploy hits one cell before the next gate; capacity — cells are sized identically and added like machines, turning regional scaling into stamping (1.11's infrastructure-as-code, again); chaos — 3.7 will kill cells on purpose; re-homing — 3.2 moves cities between cells across regions with the same lease machinery.

flowchart TB
  RT["Thin cell router — city to cell map, deliberately dumb"]
  RT --> C1
  RT --> C2
  RT --> C3
  subgraph C1 ["Cell A1 — 30 cities"]
    S1["Full T0/T1 stack"]
    D1[("Own shards, own caches, own dispatch")]
  end
  subgraph C2 ["Cell A2 — 30 cities"]
    S2["Full stack"]
    D2[("Own state")]
  end
  subgraph C3 ["Cell A3 — canary cell, quiet cities"]
    S3["Deploys land here first"]
    D3[("Own state")]
  end
  C1 -.-> K[["Shared async planes only — event log, observability, lakehouse"]]
  C2 -.-> K
  C3 -.-> K
Cells inside one region. The router above them holds almost no logic — anything smart enough to fail interestingly lives inside a cell, where its failure is bounded. Note the canary cell: 2.8's 1% is now a physical place.

Cells rot toward sharing: someone adds a "small" shared cache, a cross-cell call for a feature, a common config endpoint — and the blast-radius guarantee silently dies. Enforce isolation mechanically (no network route between cells, separate credentials) and test it in gamedays (3.7): kill a cell, verify its siblings' SLIs don't so much as twitch.

Next step

See what actually stuck.

Take the practice scenarios now.