Static stability: the data plane outlives the control plane
Foreshadowed twice: flags fail static (2.8), surge holds last-known-good (2.6). Static stability elevates that instinct into a defining property of five-nines systems.
The obvious design is to ask the control plane for current configuration or ownership on every request so the data plane is never stale. It makes the happy path simple, but it protects the wrong property. The ride-serving data plane continues from validated local state when control services are dark and uses a predefined conservative recovery for later failures.
Split every component's traffic in two. The data plane touches users: routing pings, making offers, committing transitions. The control plane configures the data plane: deploys, flag changes, rate-card updates, autoscaling decisions, lease assignment, the city→home map. The five-nines rule: the data plane must keep serving — at last-known-good configuration — through total control-plane failure. Concretely: every service caches its flags, rate cards, routing maps, and shard leases locally with generous validity; when the flag service, config store, or coordination service goes dark, nothing changes — no deploys, no re-balancing, no surge updates, and no user impact. The system freezes in its last valid shape, which is a perfectly good shape, and pages humans to fix the control plane at T2 urgency.
Why this ranks so high: mature systems rarely die of hardware — they die of their own machinery. The config push that NULLs a routing table, the coordination service election storm that revokes every lease at once, the autoscaler that scales to zero on bad metrics. Static stability inverts the dependency: control planes propose, data planes adopt — validated, versioned, rollback-able — and adoption failure means keep-what-you-have. Note the deliberate echo of CLT: just as a lesson shouldn't collapse when one prerequisite is shaky, the data plane shouldn't collapse when its instructor goes silent. Both are designs for graceful degradation of guidance.
flowchart TB
subgraph CP ["Control plane — may fail for hours"]
DEP["Deploy pipeline"]
FLAG["Flag service"]
CFG["Config and rate cards"]
COORD["Coordination — leases, home map"]
end
subgraph DP ["Data plane — must never stop"]
GWD["Gateways"]
DISPD["Dispatch"]
TRIPD["Trip state"]
LOCD["Location"]
CACHE[("Local last-known-good: flags, rates, maps, leases — versioned, validated")]
end
CP -- "propose new config (signed, versioned)" --> CACHE
CACHE -- "validate, adopt or keep current" --> DP
CPX["💥 Entire control plane dark"] -.-> CP
DP --> OK["Users unaffected — system frozen in last good shape, humans paged at leisure"]
Hidden control-plane dependencies sneak in as innocence itself: a DNS lookup per connection, a token validation that calls home, a startup that blocks on the config service — meaning you can't launch new capacity during a control-plane outage, which is precisely when autoscaling wants to. Test by starting a fresh instance with the control plane blackholed; it must come up serving from cached artifacts.