Resilience patterns: timeouts, breakers, shedding, degradation
From 1.14: vendor brownouts and traffic spikes are unhealed. From 2.1: the tier table says who gets sacrificed first. From 1.8's pitfall you've seen degradation once — complete the trip, capture later. Resilience policy generalizes that choice across the system.
A first pass might retry dependencies until they recover because returning an error seems worse than waiting. That works only while the failure case is absent. Optional dependency failure and overload must consume bounded resources, preserve critical goodput, and return an explicit fallback or refusal.
Four mechanical rules on every remaining synchronous edge. Timeouts: every call has one, shorter than the caller's own deadline, so waiting can't stack up the chain. Retries: only on idempotent operations, exponential backoff with jitter, and a retry budget (e.g. retries ≤ 10% of calls) — because a 3× retry storm is how a 30-second brownout becomes a self-inflicted outage. Circuit breakers: after enough failures, stop calling for a while and fail fast to the fallback; half-open probes test recovery. Bulkheads: per-dependency connection/thread pools, so a drowning maps vendor can't consume the threads that payments needs.
stateDiagram-v2
[*] --> Closed
Closed --> Open : failure rate over threshold in window
Open --> HalfOpen : cool-down elapsed
HalfOpen --> Closed : probe requests succeed
HalfOpen --> Open : probe fails
note right of Closed : normal — calls flow, failures counted
note right of Open : fail fast to fallback — no calls, dependency gets air
note right of HalfOpen : few real requests test recovery
Overload is the other half. When demand exceeds capacity (concert lets out, autoscaler still reacting), the worst response is treating all requests equally — everything times out, goodput hits zero, and retries pile on. Instead: load shedding by tier at the gateway and in each service — under pressure, reject T3 and T2 work with fast, honest errors and keep T0/T1 goodput high; and graceful degradation inside features — the rider map shows fewer animated cars (1.4's throttle tightens), quotes fall back to cached cell ETAs (1.3), surge holds its last-known value (1.5), receipts render late (2.4). Each flow declares its degraded mode in advance; improvising degradation during an incident is how incidents get worse.
Worked example: the maps vendor browns out
sequenceDiagram
autonumber
participant R as Riders (5,000 quotes/s)
participant P as Pricing service
participant G as Geo facade
participant B as Breaker (geo to vendor)
participant V as Maps vendor
R->>P: quote requests
P->>G: ETA lookups
G->>V: cache misses only (~10%)
V--xG: p99 30 s, errors 40%
G->>G: timeouts fire at 800 ms — no thread pileup (bulkhead holds)
B->>B: failure threshold crossed — OPEN
G->>G: fallback: serve stale cell ETAs + historical speed model
G-->>P: ETA (degraded, flagged)
P-->>R: quotes flow — slightly wider ETA ranges shown
Note over B,V: 30 s later — half-open probe: still failing → stay open
B->>V: probe succeeds after vendor recovers
B->>B: CLOSED — fresh ETAs resume
Note over R,V: rider impact: ETA precision dipped. Ride requests: 100% served. Vendor SLA decoupled from ours.
Retries multiply through layers: gateway ×3 → service ×3 → client ×3 = 27 attempts per user action. Retry at one layer (usually the caller closest to the user), budget it, and propagate deadlines so a request abandoned upstream isn't still being lovingly retried downstream.