Observability v2: SLO-driven operations
- Replace threshold alerts with multi-window burn-rate alerts tied to the error budget.
- Run the incident machine: roles, severity, and postmortems that convert incidents into architecture.
From 1.12: symptom alerts and runbooks. From 2.1: SLOs and budgets now exist — alerting can finally be about them. A 1% error threshold that was fine at three nines would let four-nines budgets burn 100× over before firing.
Burn rate = how fast you're spending error budget relative to plan (burn 1 = exactly spending a 30-day budget in 30 days). Alert when the budget is actually threatened, using two windows per alert so it's both fast and un-flappable: page when burn ≥ 14 over the last hour and the last 5 minutes (a fast, real fire — budget gone in ~2 days); ticket when burn ≥ 2 over 24 hours (a slow leak). This scheme detects a total outage in about a minute yet stays silent through the blips that threshold alerts scream about — precision and recall, both, and paging frequency becomes proportional to user harm, which is the property that keeps on-call trust alive.
The human half becomes a machine with roles: an incident commander who coordinates and decides (like 2.7's evacuation call), an ops lead who executes, a comms lead who updates status pages and execs — because the scarcest resource in an incident is the responder's working memory, and structure is how you protect it (CLT applies to operators too, at 3 a.m., under stress). Severity maps to burn rate; postmortems are blameless and produce architectural action items, each traceable to a lesson in this course: "add a breaker" (2.6), "shard was too big" (2.5), "gate lacked a business SLI" (2.8). An incident whose postmortem only produces "be more careful" will repeat.
flowchart TB
SLI["Flow SLIs at the edge — 2.1"] --> BR["Burn-rate engine"]
BR --> FAST{"burn ≥ 14 over 1 h AND 5 m?"}
BR --> SLOW{"burn ≥ 2 over 24 h?"}
FAST -- "yes" --> PAGE["Page — SEV based on burn"]
SLOW -- "yes" --> TICKET["Ticket — fix this week"]
PAGE --> IC["Incident commander + ops lead + comms lead"]
IC --> TOOLS["Traces, logs, deploy timeline, flag audit — one incident view"]
IC --> ACT["Mitigate first: rollback, flag off, shed, evacuate"]
ACT --> PM["Blameless postmortem"]
PM --> ARCH["Architecture backlog — funded by error budget policy"]
ARCH --> SLI
Watch the watchers: if the metrics pipeline dies, burn-rate alerts go silent precisely when you're blind. Meta-monitoring — a heartbeat alert that fires on telemetry absence — plus an observability stack that shares no fate with production (separate accounts, separate regions) is mandatory at this tier.