Observability v3: probes, client truth, and auto-remediation
- Measure from outside-in: synthetic probes and client telemetry become the primary SLIs.
- Wire detection to remediation for the failures whose response is mechanical.
From 2.9: burn-rate alerts on edge SLIs. At five nines the math turns brutal: a 26-second monthly budget means a one-minute detection delay is double the budget. And 3.6 created a blind spot: outages the client masks never reach server SLIs at full strength.
Detection moves outside the system. Synthetic probes — headless rider and driver clients running from every region and major mobile network — continuously execute the true T0 journey against production: go online, stream pings, request (a flagged, unmatched test ride in a reserved test city cell), receive offer, transition states. They measure what users experience, including everything server metrics can't see: DNS, TLS, edge routing, app-layer serialization. Client telemetry is the second outside-in source: apps report their own SLIs (time-to-offer, reconnect counts, hedge rates, offline-queue flushes — 3.6's check answered), aggregated with care for battery and privacy. Server-side burn rates (2.9) remain for diagnosis; probes and clients own the pager for T0.
Then close the loop where the response is mechanical. The 2.9 incident machine assumed a human mitigates; at five nines, the failures with known mitigations get auto-remediation: probe failures localized to one cell → auto-drain the cell (3.3 makes this cheap — siblings absorb its cities); localized to one region → auto re-home (3.2); localized to one deploy → auto-rollback (2.8 already does this). Humans are paged simultaneously — to supervise, and to handle the case automation misjudged. The design constraint on every auto-remediator: its worst-case action must be safe (draining a healthy cell = mild capacity loss), it must act on corroborated signals (multiple probes, multiple vantage points — one flaky probe network must not drain cells), and it must rate-limit itself (a remediator in a loop is a new failure mode; the 3.2 damping rule, generalized).
sequenceDiagram
autonumber
participant PR as Probes (multi-vantage)
participant CT as Client telemetry
participant COR as Correlator
participant REM as Auto-remediator
participant CELL as Cell A2
participant OC as On-call (supervising)
PR->>COR: cell A2 probes: offer latency 8× baseline, 3 vantage points
CT->>COR: reconnect spikes from A2's cities — corroborated
COR->>REM: verdict: cell-local degradation, high confidence
REM->>REM: guardrails — action safe? rate limit ok? scope bounded?
REM->>CELL: drain — re-home A2's cities to siblings
REM->>OC: page: acted, here's why, here's the undo
Note over CELL: users: ~15 s of reconnects. Budget spent: seconds, not minutes.
OC->>CELL: diagnose offline — cell out of rotation, no time pressure
Auto-remediation inherits the false-positive problem that made 2.3 demand quorum detection — but with bigger levers. Every remediator needs a global concurrency brake: if three "independent" cell-drain decisions fire within minutes, something systemic is being misread (probe infrastructure down? correlator bug?) — freeze, page, and let 2.9's humans take it. The machine handles the failures it was taught; the compound and the novel stay human.