Resilience patterns: timeouts, breakers, shedding, degradation
- Give every synchronous edge a budget: timeout, bounded jittered retries, circuit breaker, bulkhead.
- Design overload behavior on purpose: shed by tier, degrade features before failing flows.
From 1.14: vendor brownouts and traffic spikes are unhealed. From 2.1: the tier table says who gets sacrificed first. From 1.8's pitfall you've seen degradation once — complete the trip, capture later. This lesson makes that instinct a system-wide policy.
Four mechanical rules on every remaining synchronous edge. Timeouts: every call has one, shorter than the caller's own deadline, so waiting can't stack up the chain. Retries: only on idempotent operations, exponential backoff with jitter, and a retry budget (e.g. retries ≤ 10% of calls) — because a 3× retry storm is how a 30-second brownout becomes a self-inflicted outage. Circuit breakers: after enough failures, stop calling for a while and fail fast to the fallback; half-open probes test recovery. Bulkheads: per-dependency connection/thread pools, so a drowning maps vendor can't consume the threads that payments needs.
stateDiagram-v2
[*] --> Closed
Closed --> Open : failure rate over threshold in window
Open --> HalfOpen : cool-down elapsed
HalfOpen --> Closed : probe requests succeed
HalfOpen --> Open : probe fails
note right of Closed : normal — calls flow, failures counted
note right of Open : fail fast to fallback — no calls, dependency gets air
note right of HalfOpen : few real requests test recovery
Overload is the other half. When demand exceeds capacity (concert lets out, autoscaler still reacting), the worst response is treating all requests equally — everything times out, goodput hits zero, and retries pile on. Instead: load shedding by tier at the gateway and in each service — under pressure, reject T3 and T2 work with fast, honest errors and keep T0/T1 goodput high; and graceful degradation inside features — the rider map shows fewer animated cars (1.4's throttle tightens), quotes fall back to cached cell ETAs (1.3), surge holds its last-known value (1.5), receipts render late (2.4). Each flow declares its degraded mode in advance; improvising degradation during an incident is how incidents get worse.
Worked example: the maps vendor browns out
sequenceDiagram
autonumber
participant R as Riders (5,000 quotes/s)
participant P as Pricing service
participant G as Geo facade
participant B as Breaker (geo to vendor)
participant V as Maps vendor
R->>P: quote requests
P->>G: ETA lookups
G->>V: cache misses only (~10%)
V--xG: p99 30 s, errors 40%
G->>G: timeouts fire at 800 ms — no thread pileup (bulkhead holds)
B->>B: failure threshold crossed — OPEN
G->>G: fallback: serve stale cell ETAs + historical speed model
G-->>P: ETA (degraded, flagged)
P-->>R: quotes flow — slightly wider ETA ranges shown
Note over B,V: 30 s later — half-open probe: still failing → stay open
B->>V: probe succeeds after vendor recovers
B->>B: CLOSED — fresh ETAs resume
Note over R,V: rider impact: ETA precision dipped. Ride requests: 100% served. Vendor SLA decoupled from ours.
Retries multiply through layers: gateway ×3 → service ×3 → client ×3 = 27 attempts per user action. Retry at one layer (usually the caller closest to the user), budget it, and propagate deadlines so a request abandoned upstream isn't still being lovingly retried downstream.