fivenines
34/38
Lesson 3.6 99.999%

Client-side resilience: the app is part of the architecture

🎯 Objectives
  • Treat rider and driver apps as replicas of trip state that mask seconds of backend trouble.
  • Apply offline queueing, hedged reads, and jittered reconnection so users never see what SLIs barely record.
🔗 Connect

From 3.2: re-homing costs ~20–30 s for affected cities. From 1.7: transitions are guarded commands — which means they can be queued and validated later. From 2.6: hedging is retrying against a second server before the first fails.

The last 26 seconds of the budget are won on the phone. Three techniques. Local trip replica: both apps hold the current trip state and enough context (driver position via direct proximity of the route, fare basis, pickup point) to keep functioning through a backend gap — the map keeps animating from last-known trajectory, the driver can still tap "start trip." Offline command queue: driver-app transitions are enqueued locally with timestamps and GPS evidence, and replayed when connectivity returns; the state machine's guards (1.7) validate the replayed sequence server-side, so a trip that started, progressed, and ended inside a parking-garage dead zone reconciles perfectly at exit. The trip happened — the phone is a witness, and the server is a judge who accepts late testimony. Hedged reads + jittered reconnect: reads of trip state race a second attempt (to another cell edge or region) after p95 latency; reconnections back off with jitter so a million phones returning after re-homing don't synchronize into a thundering herd (2.2's lesson, now at planetary scale).

sequenceDiagram
    autonumber
    participant D as Driver app (in dead zone)
    participant Q as Local command queue
    participant T as Trip service
    participant R as Rider app
    Note over D: connectivity lost — trip continues physically
    D->>Q: start trip {ts, gps} — queued
    D->>Q: pings — trail buffered locally
    D->>Q: end trip {ts, gps} — queued
    Note over D: connectivity returns
    D->>T: replay queue in order
    T->>T: guards validate each: states legal, GPS plausible, ts monotonic
    T->>T: transitions applied — fare from buffered trail (1.7)
    T-->>R: trip completed — receipt flows (2.4)
    Note over T: implausible replay (teleporting GPS, fabricated ts) → flag to fraud review (2.11), not silent accept
The offline trip. Every mechanism is a reuse: guarded transitions (1.7), billing-grade trails (1.7), fraud checks on the evidence (2.11). The client is the final bulkhead.
⚠️ Pitfall

Hedging multiplies load exactly when the system is slow — which may be because of load. Hedge only idempotent reads, cap hedges to a few percent of traffic, and disable hedging under shed signals (2.6). A resilience feature with no governor becomes the incident.

Next step

See what actually stuck.

Take the practice scenarios now.