The riskiest component is change
Why, after Parts 2 and 3, your own deploys are the last big outage source standing — and the progressive-delivery pipeline that disarms them.
Cells contain failures and provide a natural canary gradient. Once hardware failures recover automatically, change becomes the largest remaining source of correlated outages.
A natural first design is to deploy globally after unit tests, choose the busiest cell as the first canary, and require human approval before rollback. But every code, configuration, and flag change earns wider exposure through production evidence; automated gates stop or reverse the ramp, and the highest-risk cell is never first.
Follow the logic of the course to its end: hardware fails → replicated (2.2, 2.8). Networks drop → healed (2.3). Overload → shed (2.7). Regions burn → survived (3.3). Sickness → detected (3.5). What remains? The one event that, by design, alters all replicas' behavior at once: a change you chose to make. Industry postmortem data has said it for decades: the majority of serious outages in mature systems are triggered by a change — code, config, or capacity. A bad build doesn't fail over, because every replica runs it faithfully (0.3's correlation warning, 2.2's poison caveat — this is their final form). The 2.5 pipeline was strong; five nines finishes it with three upgrades:
flowchart TB
RT["Replay test — full production history, outputs diffed (2.5)"] --> SH["Shadow — new build consumes LIVE traffic, outputs suppressed and diffed (2.2 machinery)"]
SH --> C1["Cell canary — smallest cell (3.4), real users, real money"]
C1 --> G1{"Automated gate: error-budget burn rate, latency, divergence, invariants (1.9)"}
G1 -- pass --> RAMP["Cell by cell: 1 → 3 → all, gate re-checked at each step"]
G1 -- fail --> RB["AUTOMATED rollback — no human approval to go back"]
RAMP --> DONE["Fleet converged — days, not minutes, and that's the point"]
RB --> PM["Postmortem: why did replay AND shadow miss it?"]
The three upgrades over 2.5, and why each earns its place:
- Cells as the ramp (3.4). The canary unit is no longer a standby engine but a complete small market. A bug that survives replay and shadow — behavior only real users trigger — detonates inside one cell's blast wall, charging 1/N of the budget.
- Automated gates on error-budget burn. Humans watching dashboards get bored, hopeful, and attached to their launches. The gate is a machine comparing the canary cell's burn rate, latencies, and 3.5 differential signals against the fleet — and its rollback needs no meeting. Humans set the thresholds beforehand, calm; machines enforce them during, fast.
- Config and capacity ride the same pipeline. The industry's most embarrassing outages are one-line config pushes that went everywhere in a second. Rule with no exceptions: anything that changes production behavior — risk limits (1.9), routing maps (3.4), feature flags, timeout tunings — ramps cell by cell through the same gates. A change too urgent for the pipeline is precisely the change most likely to need the rollback the pipeline provides.
And the loop closes on an old friend: rollback is only ever safe because of disciplines bought lessons ago — N/N−1 compatibility means the old build can always take over (2.5); sequenced activation events mean behavior flips are log entries that replay and roll back cleanly (2.5); determinism means the shadow diff catches divergence per event (1.7). Five-nines change management isn't a new system; it's the whole course, pointed at yourself.
A poison rule affects only a rare order pattern present in Cell G. The quiet canary may not expose it, but a cell-by-cell ramp and burn-rate gate limit the first evidence to one cell and roll back without waiting for a meeting.
After you've automated failure away, your own changes are the top outage source. Ramp everything — code, config, flags — cell by cell behind automated burn-rate gates. Rollback needs no approval; proceeding does. Reverting must be the easy direction.