The four-nines contract: audit your SPOFs

The four-nines contract: audit your SPOFs

A systematic pass over the Part 1 design marking every single point of failure — producing the work list for the rest of Part 2.

The three-nines venue can restart and replay, but single owners and human decisions still consume minutes. Four nines gives those weaknesses a concrete budget.

A natural first design is to count duplicate boxes on the architecture diagram and call the venue redundant without tracing authority, detection, promotion, capacity, or client recovery. But every trading-critical failure has an automatic bounded recovery path whose dependencies are outside the failed component and whose behavior is exercised regularly.

The method is blunt: for every component, ask "if exactly this dies at 14:02 on the busiest day, what stops, and for how long?" Applied to the Part 1 architecture:

flowchart TB
    GW["Gateways ×25 — already redundant"] --> SEQ["Sequencer — ONE per shard"]
    SEQ --> RISK["Risk + engine — ONE per shard"]
    RISK --> ME["Matching state — ONE copy in RAM"]
    ME --> MDP["Market data publisher — ONE"]
    SEQ --> J[("Journal — one writer, replicated disk")]
    ME --> OPS["Failover decision — A HUMAN"]
    classDef spof fill:#fecaca,stroke:#b91c1c,color:#450a0a
    classDef ok fill:#bbf7d0,stroke:#15803d,color:#052e16
    class GW ok
    class SEQ,RISK,ME,MDP,J,OPS spof
    

The audit yields the Part 2 work list, in dependency order:

1. Engine, risk, publisher state → run live replicas fed by the same event stream (2.2).
2. The stream itself → make message delivery loss-proof and gap-healing (2.3).
3. Client sessions → replication is invisible unless clients survive the switch (2.4).
4. Maintenance windows → every deploy must ride the redundancy instead of downtime (2.5).
5. The failover decision → automate it without ever electing two leaders (2.6).

Plus two items no audit of boxes reveals: overload (every box up, system still unusable — 2.7) and correlated infrastructure failure (all replicas in one blast radius — 2.8). A useful discipline going forward: redundancy is not a property of a diagram; it's a property of a tested failover path. An untested standby is a rumor.

A standby exists, but an operator must wake, inspect logs, change DNS, and verify the new primary. The extra machine improves repair options but does not remove a 20-minute outage from a 4.4-minute monthly budget.

Audit by asking "this exact box dies — what stops?" for every box. The slowest SPOF is the human failover decision. Redundancy counts only if the failover path is exercised.