fivenines
24/39

Guided Problem

PFX Build 20: Assemble the Four-Nines Venue

Time
25m
Level
intermediate
Artifacts
not specified
Progress0%
Lesson 2.9

The four-nines architecture, assembled

The complete 99.99% exchange in one diagram: every functional requirement placed, every availability story made explicit, and eight mechanisms carrying the load as one design.

Parts 2.1 through 2.8 supplied automatic recovery, replicated state, reliable streams, resumable sessions, safe change, fencing, overload policy, and multi-zone placement.

A plausible simplification is to verify each redundant box independently and infer that their composition automatically preserves one coherent trading history. The catch is that any one machine or zone failure leaves one fenced decision authority, a continuous client-visible sequence, and a bounded automatic recovery story for every trading requirement.

flowchart TB
    T["Traders — 500k concurrent sessions"]
    T --> LB["Load balancers"]
    subgraph EDGE["EDGE — stateless, spread ⅓ per zone (2.8), rolling deploys (2.5)"]
        OG["Order gateways ×25 — any gateway resumes any session (2.4)"]
        FS["Feed servers — L1/L2/L3 tiers + conflation (1.8), NAK repair (2.3)"]
        AG["Account and admin APIs"]
    end
    LB --> OG
    T --> AG
    subgraph CORE["TRADING CORE — one symbol shard shown, ×N (1.11), spread across zones A · B · C"]
        EL["Election group ×3 — quorum votes, epoch fencing (2.6)"]
        SEQ["Sequencer leader + pre-trade risk (1.9) + session state (2.4)"]
        ME1["Engine PRIMARY — zone A, publishes"]
        ME2["Engine standby — zone B, suppressed (2.2)"]
        ME3["Engine standby — zone C, suppressed"]
        J[("Journals ×3 zones — quorum append (2.3)")]
        RTX["Retransmit servers (2.3)"]
    end
    OG --> SEQ
    EL -.-> SEQ
    SEQ --> J
    SEQ --> ME1 & ME2 & ME3
    ME1 --> MDP["Market data publisher"] --> FS --> T
    ME1 -- acks and fills --> OG
    RTX -.-> FS
    J --> PT["POST-TRADE — clearing, netting, settlement, surveillance, audit (1.10) — async consumers, pause and catch up"]
    
The four-nines exchange. Same four jobs as lesson 0.1, same tiers as 1.1 — but every box that was red in the 2.1 SPOF audit is now a quorum, a fleet, or a suppressed twin. Trace any functional arrow and there is no longer a single machine whose death interrupts it for more than seconds.

Read the diagram as the 2.1 audit, answered. The sequencer is no longer "one process" but a leadership role that a three-node election group reassigns in under a second, with epochs making deposed leaders harmless (2.6). The engine is three deterministic replicas consuming one stream, exactly one of which publishes (2.2). The journal is three zone-separated replicas acknowledging by quorum (2.3). The gateways were always a fleet (1.1); what's new is that sessions survive any member's death because session truth lives in the core (2.4). The human is gone from the failover path entirely — the 2.8 walk-through demonstrated a zone loss handled in seconds with the on-call merely watching.

Now place every functional requirement from the 0.2 map into this skeleton and ask lesson 2.1's question of each — the table is the whole part in one view:

Functional requirementWhere it livesInstance diesZone dies
Order entry, cancel, modifyGateways → sequenced coreSession resumes on another gateway, replay proves state (2.4)Seconds of pause; reconnect to surviving zones (2.8)
Matching + order lifecycle (1.3–1.5)Engine replicas ×3Standby promoted, fenced, resumes at next event (2.2, 2.6)Same — replicas span zones by design
Pre-trade risk (1.9)In the sequenced path, replicated with itTravels with the engine failover — balances are core stateSame
Auctions, halts, phases (1.6)Events in the logNothing special — replay and replicas carry phase state like any other stateSame
Market data L1/L2/L3 (1.8)Publisher + feed fleetConsumers NAK-repair from retransmit servers (2.3)Feed capacity at ⅔ headroom absorbs it (2.8)
Clearing, settlement, audit (1.10)Async journal consumersPause, restart, catch up — invisible to tradingSame, from surviving journal replicas
Deploys of all of the above (2.5)The redundancy itselfEdge rolls; core upgrades by failover; post-trade pauses — zero scheduled downtime

One honest asterisk survives, and it matters: the engine publishes outputs the instant it processes an event, while journal quorum acknowledgment completes microseconds-to-milliseconds later. There is a hair's-width window in which a published fill could outrun its own durability. At four nines we accept it; closing it is the first act of Part 3 (3.2).

The engine publishes a fill and dies before the journal quorum stores its causal event. Every failover mechanism works, yet replay omits a trade a client already saw. Integration exposes this last durability gap because no individual redundant box reveals it.

Four nines = the 1.1 functional architecture with every SPOF replaced by quorum, fleet, or twin. Every functional requirement now has a per-failure story measured in seconds, humans excluded. Remaining gap: outputs can outrun durability by a hair — Part 3's opening move.

Next step

See what actually stuck.

Take the practice scenarios now.