Observability v3: probes, client truth, and auto-remediation
From 2.9: burn-rate alerts on edge SLIs. At five nines the math turns brutal: a 26-second monthly budget means a one-minute detection delay is double the budget. And 3.6 created a blind spot: outages the client masks never reach server SLIs at full strength.
The simplest version would remediate any alarming metric automatically because machines react faster than operators. The better question is what survives when that assumption fails. Automatic action requires corroborated user evidence, a bounded target, spare capacity, an immediate undo, and post-action verification.
Detection moves outside the system. Synthetic probes — headless rider and driver clients running from every region and major mobile network — continuously execute the true T0 journey against production: go online, stream pings, request (a flagged, unmatched test ride in a reserved test city cell), receive offer, transition states. They measure what users experience, including everything server metrics can't see: DNS, TLS, edge routing, app-layer serialization. Client telemetry is the second outside-in source: apps report their own SLIs (time-to-offer, reconnect counts, hedge rates, offline-queue flushes — 3.6's check answered), aggregated with care for battery and privacy. Server-side burn rates (2.9) remain for diagnosis; probes and clients own the pager for T0.
Then close the loop where the response is mechanical. The 2.9 incident machine assumed a human mitigates; at five nines, the failures with known mitigations get auto-remediation: probe failures localized to one cell → auto-drain the cell (3.3 makes this cheap — siblings absorb its cities); localized to one region → auto re-home (3.2); localized to one deploy → auto-rollback (2.8 already does this). Humans are paged simultaneously — to supervise, and to handle the case automation misjudged. The design constraint on every auto-remediator: its worst-case action must be safe (draining a healthy cell = mild capacity loss), it must act on corroborated signals (multiple probes, multiple vantage points — one flaky probe network must not drain cells), and it must rate-limit itself (a remediator in a loop is a new failure mode; the 3.2 damping rule, generalized).
sequenceDiagram
autonumber
participant PR as Probes (multi-vantage)
participant CT as Client telemetry
participant COR as Correlator
participant REM as Auto-remediator
participant CELL as Cell A2
participant OC as On-call (supervising)
PR->>COR: cell A2 probes: offer latency 8× baseline, 3 vantage points
CT->>COR: reconnect spikes from A2's cities — corroborated
COR->>REM: verdict: cell-local degradation, high confidence
REM->>REM: guardrails — action safe? rate limit ok? scope bounded?
REM->>CELL: drain — re-home A2's cities to siblings
REM->>OC: page: acted, here's why, here's the undo
Note over CELL: users: ~15 s of reconnects. Budget spent: seconds, not minutes.
OC->>CELL: diagnose offline — cell out of rotation, no time pressure
Auto-remediation inherits the false-positive problem that made 2.3 demand quorum detection — but with bigger levers. Every remediator needs a global concurrency brake: if three "independent" cell-drain decisions fire within minutes, something systemic is being misread (probe infrastructure down? correlator bug?) — freeze, page, and let 2.9's humans take it. The machine handles the failures it was taught; the compound and the novel stay human.