State at five nines: quorum truth, ephemeral speed
- Give trip state a quorum-replicated store that survives node and zone loss with zero failover pause.
- Formalize the two-class state doctrine: quorum for truth, rebuildable memory for speed, nothing in between.
From 2.3: automated failover still has a ~20 s write pause — 20 s is most of a five-nines monthly budget. From 1.4/2.5: location state needs no durability at all. From 1.7: trip state is the record money and safety hang from. The extremes are solved; this lesson finishes the middle.
For T0's durable core — trip states and dispatch assignments — primary/standby is retired in favor of a quorum-replicated store (Raft/Paxos-class): five replicas per shard across zones, every write committed by majority (3 of 5), reads served consistently from the leader or by quorum. When a node or zone dies there is no failover event — the replica group elects a leader in sub-second time and writes continue; the 20-second pause from 2.3 simply doesn't exist. The cost is honest: quorum writes add a few milliseconds, capacity multiplies, and operations get subtler — which is exactly why 3.1 confined this machinery to the flows that earn it. The conditional assignment of 1.6 — the marketplace's oldest invariant — now executes as a quorum write: the linearizable heart of the exchange.
This completes a doctrine worth stating as a law: state is either quorum-replicated truth or rebuildable ephemera — never "important but on one box." Audit every byte of T0 against it: trip state → quorum. Assignments → quorum. Location index → ephemeral, rebuilt in 4 s. Dispatch market memory → ephemeral, rebuilt by lease claim. Surge values → ephemeral with last-known-good hold (2.6). Quotes → T1, replicated conventionally; a lost quote re-quotes. And when even the quorum store suffers (network partition inside a cell), the state machine's guards (1.7) plus client-side queueing (3.6) let trips buffer transitions briefly rather than error — degrade, never fail, all the way down.
flowchart TB
subgraph QG ["Trip-state quorum group — one per shard, 5 replicas across zones"]
L["Leader — zone A"]
F1["Follower — zone A"]
F2["Follower — zone B"]
F3["Follower — zone B"]
F4["Follower — zone C"]
end
W["Trip transition write"] --> L
L --> F2
L --> F3
L -. "async catch-up" .-> F1
L -. " " .-> F4
L --> ACK["Committed at 3 of 5 — few ms"]
ZL["Zone B lost — 2 replicas gone"] -.-> QG
QG --> OK["3 of 5 remain — quorum holds, zero pause, no failover event"]
Quorum systems fail differently, not never: lose the majority (two zones, or a partition isolating the leader with one follower) and the group stops accepting writes — correctly, to protect consistency. Your design must decide in advance what dispatch does during a minority partition: serve stale reads, queue transitions with client cooperation, or shed that cell and re-home its cities (3.2). "The store is CP, so we're done" is half a design.