The hot standby: replicated state machines
How determinism turns replication from a hard problem into a subscription — and what failover looks like when the standby is already caught up.
Deterministic replay already recreates engine state. A continuously replaying subscriber can therefore remain close enough to take ownership in seconds.
It is tempting to let both engines publish for convenience, or promote the standby after one missed heartbeat without first revoking the old primary's authority. The constraint it misses is simple: Both engines consume the same sequence, exactly one fenced epoch may publish, and promotion resumes at the first event not already made visible.
Replicating a mutable database is painful: which writes have propagated? Are replicas transactionally consistent? Our core sidesteps all of it. The engine is a deterministic function of the log — so a replica is just a second engine consuming the same log. No state synchronization protocol, no diffing: identical inputs in identical order produce identical state, every event, by construction.
sequenceDiagram
autonumber
participant S as Sequenced stream
participant P as Primary engine
participant B as Standby engine
participant F as Failover controller
S->>P: event 1041
S->>B: event 1041
P->>P: process, PUBLISH outputs
B->>B: process identically, SUPPRESS outputs
Note over P: primary dies after event 1041
F->>F: heartbeat missing ×3 (~600 ms)
F->>P: fence — old primary may no longer publish
F->>B: promote
B->>B: confirm processed through 1041
B->>S: publishing resumes from event 1042
The three rules that make this correct:
- Exactly one publisher. Both replicas compute everything; only the primary's outputs (fills, market data) leave the building. Two publishers would double-send every fill.
- Fence before promote. The old primary might be alive-but-slow rather than dead (the gray failure — 3.5). Before the standby publishes, the old primary must be verifiably cut off — downstream systems reject its epoch (2.6 formalizes this).
- Resume, don't rewind. The new primary continues from the exact event where outputs stopped. Consumers see a pause, never a repeat and never a gap — the sequence numbers guarantee they can prove it.
Cost accounting, honestly: 2× core compute (a rounding error next to venue revenue), plus one sharp new risk — correlated software failure. If event 1041 is a poison message that crashes the primary via a matching bug, it crashes the standby identically one instant later, because determinism replicates bugs with the same fidelity as state. Lesson 0.3's warning made concrete. Mitigations: run standbys one software version behind during rollouts (2.5 exploits this deliberately), aggressive input validation at the sequencer, and a rehearsed break-glass path back to restart-and-replay. Determinism giveth; determinism replicateth the bug.
The primary loses contact with the standby but still serves gateways. If the standby promotes on timeout alone, both can emit fills from diverging input. Output suppression plus fencing makes “caught up” insufficient without exclusive authority.
A replica = another subscriber to the log. Determinism does the synchronization. One publisher; fence the old before promoting the new; resume exactly. Determinism replicates bugs too — version-stagger your replicas.