Gray failures and chaos drills

Every mechanism since 2.2 triggers on a binary: heartbeats stop, quorum lost, process dead. But mature systems mostly don't die — they sicken. A NIC drops 2% of packets. A disk's writes take 50× longer. A feed server serves quotes 800 ms stale. Every health check passes; every SLO burns. This is gray failure, and it's the dominant incident class at five nines for a structural reason: the binary failures all got automated away in Part 2, and what remains is precisely what binary detectors can't see. Worse, an exchange's gray failures are unfair before they're visible — the clients on the sick feed server trade on stale prices against clients on healthy ones.

Detection: stop asking components how they feel; compare them to their peers. The defining property of gray failure is disagreement between perspectives — the component says healthy, its users experience sick. So instrument the disagreement itself:

flowchart TB
    M["Per-replica, per-path metrics: p99 latency, staleness, gap-request rate, output digests"] --> CMP{"Peer comparison — is one replica an outlier vs its siblings?"}
    CMP -- "all agree" --> OK["Healthy — including 'all equally slow', which is load (2.7), not sickness"]
    CMP -- "one disagrees" --> ACT["Demote the outlier — drain traffic, drop from quorum preference, page for autopsy"]
    ACT --> KEY["Key insight: with 3+ peers, 'who is lying?' becomes a majority vote — the quorum idea (2.6), applied to health"]
    

Verification: a failover path you haven't run this quarter doesn't exist. Part 3's mechanisms are mostly rare paths — and rare paths rot: configs drift, capacity assumptions expire, runbooks reference decommissioned hosts. The counter-practice is drilling, in production, on a calendar:

• Continuous, automated: kill a random core replica weekly (verify 2.2/2.6 failover), inject packet loss on a feed path (verify 2.3 gap-fill and the differential detectors above), restart post-trade mid-day (verify 1.10's indifference).
• Scheduled, human: quarterly zone evacuation (the 2.8 walk-through, on purpose); annual full region failover with real client reconnection (3.3's runbook, rehearsed until boring — the drill exists to make the real thing a rerun).
• Gray-failure injection, the honest exam: don't kill the replica — slow it. Add 20 ms to one journal's writes; make one feed server stale by 500 ms. Binary detectors will certify it healthy; only the differential layer can catch it. This is the test most systems fail on the first run, which is precisely the argument for running it.

Cells (3.4) make all of this affordable: drills run against one cell, bounding the worst case to one-Nth of the market — which is why mature five-nines shops inject failure into production weekly yet appear, from outside, to simply never fail. The causality runs the unintuitive direction: they're reliable because they break themselves constantly, on their own schedule, while everyone's watching — instead of on the market's schedule, while no one is.