Observability v1: golden signals and on-call
- Stand up the three telemetry pillars — metrics, logs, traces — and alert on symptoms, not causes.
- Define the golden signals per service and the on-call loop that turns alerts into fixes.
From 0.3's pitfall: "down" means riders can't ride, not servers are sad. Observability exists to measure the system in the rider's terms. From 1.11: humans handle the big failures — this lesson is how humans find out.
Every service emits the same three streams through the same agents: metrics (time series: rate, errors, duration histograms — the RED signals — plus saturation), structured logs (JSON events with trip/rider/driver ids), and distributed traces (every request carries a trace id from the gateway through every hop — indispensable when a slow quote might be gateway, pricing, geo facade, or the vendor). The discipline that matters more than the tooling: alert on symptoms, diagnose with causes. Page a human only when users are affected — "ride requests failing > 1%", "p99 quote latency > 2 s", "dispatch offers not reaching drivers" — and let cause-level signals (CPU, queue depth, replication lag) be dashboards and warnings that inform diagnosis, not pages. Symptom alerts are few, stable, and trustworthy; cause alerts multiply forever and train on-call to ignore them.
flowchart LR
subgraph SVCS ["Every service and host"]
AG["Telemetry agents — metrics, logs, trace spans"]
end
AG --> M[("Metrics store — time series")]
AG --> LG[("Log store — searchable, 30 d hot")]
AG --> TR[("Trace store — sampled")]
M --> DASH["Dashboards — per service + business overview"]
M --> AL["Alert rules — symptom-based"]
AL --> PD["Pager — on-call rotation"]
PD --> OC["👩💻 On-call engineer"]
OC --> RB["Runbooks — linked from every alert"]
OC --> DASH
OC --> LG
OC --> TR
Worked example: a Saturday-night page
sequenceDiagram
autonumber
participant M as Metrics store
participant A as Alert engine
participant P as Pager
participant E as On-call engineer
participant DB as PostgreSQL
M->>A: ride-request error rate 3% over 5 min (threshold 1%)
A->>P: fire page — SYMPTOM: requests failing
P->>E: page + runbook link
E->>E: dashboard: errors only on writes, reads fine
E->>DB: replication lag panel — standby healthy, primary disk 98%
E->>DB: purge bloated temp files, extend volume
M->>A: error rate back under 0.1%
A->>P: auto-resolve
E->>E: file incident note — feeds postmortem and 1.14 recap
Unbounded label cardinality — a metric labeled per rider id or per trip id — will melt the metrics store. Ids belong in logs and traces; metrics get bounded labels (service, city, tier, status). This one rule saves the observability bill and the observability availability.