fivenines
30/39
Lesson 3.6

The riskiest component is change

What you'll learn

Why, after Parts 2 and 3, your own deploys are the last big outage source standing — and the progressive-delivery pipeline that disarms them.

Follow the logic of the course to its end: hardware fails → replicated (2.2, 2.8). Networks drop → healed (2.3). Overload → shed (2.7). Regions burn → survived (3.3). Sickness → detected (3.5). What remains? The one event that, by design, alters all replicas' behavior at once: a change you chose to make. Industry postmortem data has said it for decades: the majority of serious outages in mature systems are triggered by a change — code, config, or capacity. A bad build doesn't fail over, because every replica runs it faithfully (0.3's correlation warning, 2.2's poison caveat — this is their final form). The 2.5 pipeline was strong; five nines finishes it with three upgrades:

flowchart TB
    RT["Replay test — full production history, outputs diffed (2.5)"] --> SH["Shadow — new build consumes LIVE traffic, outputs suppressed and diffed (2.2 machinery)"]
    SH --> C1["Cell canary — smallest cell (3.4), real users, real money"]
    C1 --> G1{"Automated gate: error-budget burn rate, latency, divergence, invariants (1.9)"}
    G1 -- pass --> RAMP["Cell by cell: 1 → 3 → all, gate re-checked at each step"]
    G1 -- fail --> RB["AUTOMATED rollback — no human approval to go back"]
    RAMP --> DONE["Fleet converged — days, not minutes, and that's the point"]
    RB --> PM["Postmortem: why did replay AND shadow miss it?"]
    
Progressive delivery over cells. Each stage multiplies exposed risk by a small factor and is watched by an automated gate. Note the asymmetry: proceeding requires passing a gate; rolling back requires nothing — reverting must always be the easy direction.

The three upgrades over 2.5, and why each earns its place:

  • Cells as the ramp (3.4). The canary unit is no longer a standby engine but a complete small market. A bug that survives replay and shadow — behavior only real users trigger — detonates inside one cell's blast wall, charging 1/N of the budget.
  • Automated gates on error-budget burn. Humans watching dashboards get bored, hopeful, and attached to their launches. The gate is a machine comparing the canary cell's burn rate, latencies, and 3.5 differential signals against the fleet — and its rollback needs no meeting. Humans set the thresholds beforehand, calm; machines enforce them during, fast.
  • Config and capacity ride the same pipeline. The industry's most embarrassing outages are one-line config pushes that went everywhere in a second. Rule with no exceptions: anything that changes production behavior — risk limits (1.9), routing maps (3.4), feature flags, timeout tunings — ramps cell by cell through the same gates. A change too urgent for the pipeline is precisely the change most likely to need the rollback the pipeline provides.

And the loop closes on an old friend: rollback is only ever safe because of disciplines bought lessons ago — N/N−1 compatibility means the old build can always take over (2.5); sequenced activation events mean behavior flips are log entries that replay and roll back cleanly (2.5); determinism means the shadow diff catches divergence per event (1.7). Five-nines change management isn't a new system; it's the whole course, pointed at yourself.

Key ideas
  • After you've automated failure away, your own changes are the top outage source.
  • Ramp everything — code, config, flags — cell by cell behind automated burn-rate gates.
  • Rollback needs no approval; proceeding does. Reverting must be the easy direction.
Next step

See what actually stuck.

Take the practice scenarios now.