Hardening the topology
From 1.11: instances and zones already self-heal. Four nines demands more: capacity to absorb a zone loss at peak without scrambling, and ejection of unhealthy-but-alive instances — the gray failures that never trip a simple health check.
A first pass might rely on autoscaling after a zone fails and use process liveness as the only routing signal. That works only while the failure case is absent. Critical capacity must already tolerate one zone loss, and unhealthy or replaced instances must hold no irreplaceable request state.
Three upgrades. First, N+1 at the zone level: every tier is provisioned so that losing one full zone at peak leaves enough capacity — that means each of 3 zones runs at ≤ 66% of peak need. Autoscaling still runs, but the floor guarantees the zone-loss case; scaling is for economics, floors are for availability. Second, health beyond liveness: load balancers and the service mesh eject instances on error rate and latency (outlier detection), not just "port answers" — a process that responds slowly poisons more requests than one that's dead, because the balancer keeps feeding it. Third, statelessness audited, not assumed: session state in the token (1.2) ✓, but WebSocket connections (1.4) are state — so the realtime gateway gets connection draining and client auto-reconnect with jittered backoff, making gateway instance loss a 2-second blip instead of a mass disconnect stampede.
flowchart TB
LB["Load balancer + service mesh"] --> HC{"Health: liveness AND error rate AND latency outlier?"}
HC -- "healthy" --> POOL["Instance pool — floor: any 2 zones carry peak"]
HC -- "gray failure" --> EJECT["Eject from rotation, alert cause-level"]
EJECT --> REPLACE["Orchestrator replaces instance automatically"]
REPLACE --> POOL
subgraph SCALE ["Autoscaling"]
MET["Scale on leading signals — RPS, queue depth"] --> UP["Scale up fast"]
MET --> DOWN["Scale down slowly, small steps"]
end
SCALE --> POOL
Scaling on CPU alone misses the failure that matters: threads blocked on a slow dependency burn no CPU while the service drowns. Scale on request rate and queue depth (leading), keep CPU as a backstop (lagging). And cap ejection — if the "outlier detector" wants to eject 40% of a pool, the problem is shared (a dependency, a deploy), and ejecting amplifies it.