fivenines
16/39
Lesson 2.1

The four-nines contract: audit your SPOFs

What you'll learn

A systematic pass over the Part 1 design marking every single point of failure — producing the work list for the rest of Part 2.

The method is blunt: for every component, ask "if exactly this dies at 14:02 on the busiest day, what stops, and for how long?" Applied to the Part 1 architecture:

flowchart TB
    GW["Gateways ×25 — already redundant"] --> SEQ["Sequencer — ONE per shard"]
    SEQ --> RISK["Risk + engine — ONE per shard"]
    RISK --> ME["Matching state — ONE copy in RAM"]
    ME --> MDP["Market data publisher — ONE"]
    SEQ --> J[("Journal — one writer, replicated disk")]
    ME --> OPS["Failover decision — A HUMAN"]
    classDef spof fill:#fecaca,stroke:#b91c1c,color:#450a0a
    classDef ok fill:#bbf7d0,stroke:#15803d,color:#052e16
    class GW ok
    class SEQ,RISK,ME,MDP,J,OPS spof
    
The SPOF audit. Green survives instance loss today; red means minutes of downtime per event. The most dangerous box isn't a machine at all — it's the human in the failover decision.

The audit yields the Part 2 work list, in dependency order:

  1. Engine, risk, publisher state → run live replicas fed by the same event stream (2.2).
  2. The stream itself → make message delivery loss-proof and gap-healing (2.3).
  3. Client sessions → replication is invisible unless clients survive the switch (2.4).
  4. Maintenance windows → every deploy must ride the redundancy instead of downtime (2.5).
  5. The failover decision → automate it without ever electing two leaders (2.6).

Plus two items no audit of boxes reveals: overload (every box up, system still unusable — 2.7) and correlated infrastructure failure (all replicas in one blast radius — 2.8). A useful discipline going forward: redundancy is not a property of a diagram; it's a property of a tested failover path. An untested standby is a rumor.

Key ideas
  • Audit by asking "this exact box dies — what stops?" for every box.
  • The slowest SPOF is the human failover decision.
  • Redundancy counts only if the failover path is exercised.
Next step

See what actually stuck.

Take the practice scenarios now.