The hot standby: replicated state machines

Replicating a mutable database is painful: which writes have propagated? Are replicas transactionally consistent? Our core sidesteps all of it. The engine is a deterministic function of the log — so a replica is just a second engine consuming the same log. No state synchronization protocol, no diffing: identical inputs in identical order produce identical state, every event, by construction.

sequenceDiagram
    autonumber
    participant S as Sequenced stream
    participant P as Primary engine
    participant B as Standby engine
    participant F as Failover controller
    S->>P: event 1041
    S->>B: event 1041
    P->>P: process, PUBLISH outputs
    B->>B: process identically, SUPPRESS outputs
    Note over P: primary dies after event 1041
    F->>F: heartbeat missing ×3 (~600 ms)
    F->>P: fence — old primary may no longer publish
    F->>B: promote
    B->>B: confirm processed through 1041
    B->>S: publishing resumes from event 1042
    

The three rules that make this correct:

• Exactly one publisher. Both replicas compute everything; only the primary's outputs (fills, market data) leave the building. Two publishers would double-send every fill.
• Fence before promote. The old primary might be alive-but-slow rather than dead (the gray failure — 3.5). Before the standby publishes, the old primary must be verifiably cut off — downstream systems reject its epoch (2.6 formalizes this).
• Resume, don't rewind. The new primary continues from the exact event where outputs stopped. Consumers see a pause, never a repeat and never a gap — the sequence numbers guarantee they can prove it.

Cost accounting, honestly: 2× core compute (a rounding error next to venue revenue), plus one sharp new risk — correlated software failure. If event 1041 is a poison message that crashes the primary via a matching bug, it crashes the standby identically one instant later, because determinism replicates bugs with the same fidelity as state. Lesson 0.3's warning made concrete. Mitigations: run standbys one software version behind during rollouts (2.5 exploits this deliberately), aggressive input validation at the sequencer, and a rehearsed break-glass path back to restart-and-replay. Determinism giveth; determinism replicateth the bug.