Hardening the topology
- Remove every human from instance/zone/service recovery: N+1 capacity, health-based ejection, safe autoscaling.
- Make every stateless service truly stateless — and find the hidden state that isn't.
From 1.11: instances and zones already self-heal. Four nines demands more: capacity to absorb a zone loss at peak without scrambling, and ejection of unhealthy-but-alive instances — the gray failures that never trip a simple health check.
Three upgrades. First, N+1 at the zone level: every tier is provisioned so that losing one full zone at peak leaves enough capacity — that means each of 3 zones runs at ≤ 66% of peak need. Autoscaling still runs, but the floor guarantees the zone-loss case; scaling is for economics, floors are for availability. Second, health beyond liveness: load balancers and the service mesh eject instances on error rate and latency (outlier detection), not just "port answers" — a process that responds slowly poisons more requests than one that's dead, because the balancer keeps feeding it. Third, statelessness audited, not assumed: session state in the token (1.2) ✓, but WebSocket connections (1.4) are state — so the realtime gateway gets connection draining and client auto-reconnect with jittered backoff, making gateway instance loss a 2-second blip instead of a mass disconnect stampede.
flowchart TB
LB["Load balancer + service mesh"] --> HC{"Health: liveness AND error rate AND latency outlier?"}
HC -- "healthy" --> POOL["Instance pool — floor: any 2 zones carry peak"]
HC -- "gray failure" --> EJECT["Eject from rotation, alert cause-level"]
EJECT --> REPLACE["Orchestrator replaces instance automatically"]
REPLACE --> POOL
subgraph SCALE ["Autoscaling"]
MET["Scale on leading signals — RPS, queue depth"] --> UP["Scale up fast"]
MET --> DOWN["Scale down slowly, small steps"]
end
SCALE --> POOL
Scaling on CPU alone misses the failure that matters: threads blocked on a slow dependency burn no CPU while the service drowns. Scale on request rate and queue depth (leading), keep CPU as a backstop (lagging). And cap ejection — if the "outlier detector" wants to eject 40% of a pool, the problem is shared (a dependency, a deploy), and ejecting amplifies it.