Safe change: canaries, flags, and reversible migrations
- Make deploys progressive and self-reverting: canary → metrics gate → staged rollout.
- Separate deploy from release with feature flags, and make schema changes backward-compatible by rule.
From 1.14: a bad deploy burns 10–30 minutes — most industry outages are self-inflicted by change, not by hardware. From 2.1: the error budget prices this. From 2.9 (ahead): the metrics gate needs trustworthy SLIs — these two lessons are a pair.
At four nines, change is the dominant risk, so the pipeline's job flips from "ship fast" to "ship reversibly". Three mechanisms. Progressive delivery: every deploy goes first to a canary slice (1% of traffic, one zone), where an automated gate compares the canary's SLIs — error rate, p99, and business signals like offer-acceptance rate — against the baseline for 15 minutes; pass → staged rollout (5% → 25% → 100%, gated at each step), fail → automatic rollback, no human required. Feature flags separate deploying code from releasing behavior: risky logic ships dark, ramps by percentage/city, and — critically — turns off in seconds when it misbehaves, which is faster than any rollback. The flag service itself must fail static (last-known-good values cached in every service; a foreshadowing of 3.5). Expand–contract migrations keep every schema change compatible with the previous code version: add the new column and dual-write (expand), backfill and switch reads, only then drop the old path (contract) — because rollback is only real if version N−1 still runs against today's schema.
sequenceDiagram
autonumber
participant CI as CI pipeline
participant CD as Progressive delivery
participant CAN as Canary (1%, one zone)
participant GATE as Metrics gate
participant FLEET as Fleet (staged)
CI->>CD: image passed tests + staging
CD->>CAN: deploy to canary slice
GATE->>GATE: 15 min: canary vs baseline — errors, p99, offer-accept rate
alt canary healthy
GATE->>FLEET: promote 5% → 25% → 100%, gate at each step
else canary degraded
GATE->>CAN: automatic rollback
GATE->>CI: block pipeline, page owner, annotate error budget spend
end
Note over GATE: business SLIs in the gate catch what error rates miss — a ranker that quietly tanks acceptance
Config and flags are deploys too — most large outages start with "it was just a config change." Route them through the same canary gate. And beware flag debt: every stale flag doubles the state space your on-call must reason about at 3 a.m.; expire them like milk.