Safe change: canaries, flags, and reversible migrations
From 1.14: a bad deploy burns 10–30 minutes — most industry outages are self-inflicted by change, not by hardware. From 2.1: the error budget prices this. From 2.9 (ahead): the metrics gate needs trustworthy SLIs — these two lessons are a pair.
The simplest version would deploy a small code canary and assume a database change can be rolled back the same way. The better question is what survives when that assumption fails. Every release stage bounds exposure and retains a tested reverse path across code, configuration, and schema versions.
At four nines, change is the dominant risk, so the pipeline's job flips from "ship fast" to "ship reversibly". Three mechanisms. Progressive delivery: every deploy goes first to a canary slice (1% of traffic, one zone), where an automated gate compares the canary's SLIs — error rate, p99, and business signals like offer-acceptance rate — against the baseline for 15 minutes; pass → staged rollout (5% → 25% → 100%, gated at each step), fail → automatic rollback, no human required. Feature flags separate deploying code from releasing behavior: risky logic ships dark, ramps by percentage/city, and — critically — turns off in seconds when it misbehaves, which is faster than any rollback. The flag service itself must fail static (last-known-good values cached in every service; a foreshadowing of 3.5). Expand–contract migrations keep every schema change compatible with the previous code version: add the new column and dual-write (expand), backfill and switch reads, only then drop the old path (contract) — because rollback is only real if version N−1 still runs against today's schema.
sequenceDiagram
autonumber
participant CI as CI pipeline
participant CD as Progressive delivery
participant CAN as Canary (1%, one zone)
participant GATE as Metrics gate
participant FLEET as Fleet (staged)
CI->>CD: image passed tests + staging
CD->>CAN: deploy to canary slice
GATE->>GATE: 15 min: canary vs baseline — errors, p99, offer-accept rate
alt canary healthy
GATE->>FLEET: promote 5% → 25% → 100%, gate at each step
else canary degraded
GATE->>CAN: automatic rollback
GATE->>CI: block pipeline, page owner, annotate error budget spend
end
Note over GATE: business SLIs in the gate catch what error rates miss — a ranker that quietly tanks acceptance
Config and flags are deploys too — most large outages start with "it was just a config change." Route them through the same canary gate. And beware flag debt: every stale flag doubles the state space your on-call must reason about at 3 a.m.; expire them like milk.