The four-nines contract: audit your SPOFs
What you'll learn
A systematic pass over the Part 1 design marking every single point of failure — producing the work list for the rest of Part 2.
The method is blunt: for every component, ask "if exactly this dies at 14:02 on the busiest day, what stops, and for how long?" Applied to the Part 1 architecture:
flowchart TB
GW["Gateways ×25 — already redundant"] --> SEQ["Sequencer — ONE per shard"]
SEQ --> RISK["Risk + engine — ONE per shard"]
RISK --> ME["Matching state — ONE copy in RAM"]
ME --> MDP["Market data publisher — ONE"]
SEQ --> J[("Journal — one writer, replicated disk")]
ME --> OPS["Failover decision — A HUMAN"]
classDef spof fill:#fecaca,stroke:#b91c1c,color:#450a0a
classDef ok fill:#bbf7d0,stroke:#15803d,color:#052e16
class GW ok
class SEQ,RISK,ME,MDP,J,OPS spof
The audit yields the Part 2 work list, in dependency order:
- Engine, risk, publisher state → run live replicas fed by the same event stream (2.2).
- The stream itself → make message delivery loss-proof and gap-healing (2.3).
- Client sessions → replication is invisible unless clients survive the switch (2.4).
- Maintenance windows → every deploy must ride the redundancy instead of downtime (2.5).
- The failover decision → automate it without ever electing two leaders (2.6).
Plus two items no audit of boxes reveals: overload (every box up, system still unusable — 2.7) and correlated infrastructure failure (all replicas in one blast radius — 2.8). A useful discipline going forward: redundancy is not a property of a diagram; it's a property of a tested failover path. An untested standby is a rumor.
Key ideas
- Audit by asking "this exact box dies — what stops?" for every box.
- The slowest SPOF is the human failover decision.
- Redundancy counts only if the failover path is exercised.