Operating at three nines: crash, restart, replay
What failure and recovery actually look like at this tier, arithmetic included — and precisely which weaknesses buy Part 2 its existence.
The log is authoritative, snapshots shorten recovery, and each shard has bounded ownership. Three-nines operation now needs an explicit recovery procedure.
A plausible simplification is to load the most recent snapshot, declare the process healthy, and reopen before replay reaches the durable journal head. The catch is that recovery rebuilds every shard to the committed sequence, verifies book, balance, and state-machine invariants, and only then admits new trading.
At three nines, the recovery plan for the core is honest and simple: one authoritative engine per shard; if it dies, restart it and replay. Walk the clock through an incident:
flowchart LR
A["14:02:00 — engine process dies"] --> B["14:02:15 — health check fails, page fires"]
B --> C["14:04 — on-call confirms, triggers restart runbook"]
C --> D["14:05 — snapshot loaded (5 min old)"]
D --> E["14:06 — journal replayed to head"]
E --> F["14:07 — gateways reconnect, resume"]
F --> G["~5–7 minutes downtime for this shard's symbols"]
Engineering that makes the timeline real:
- Snapshot cadence bounds replay time. Snapshot every 5 minutes; replay runs ~50× real time; worst-case replay ≈ 6 seconds. The journal and snapshots live on storage that survives the process — at this tier, a replicated disk is acceptable.
- Monitor the four golden signals per component — traffic, errors, latency, saturation — plus two exchange-specific ones: sequence-gap alarms (any consumer seeing gaps in the event stream) and invariant checks (reserved balances match live orders, ledger matches replay — from 1.9/1.10). Alert on symptoms users feel, not on every twitching metric.
- Maintenance uses the budget deliberately. Upgrades happen in scheduled low-activity windows: drain, upgrade, restart, replay, verify. Planned and boring — at this tier, downtime for change is a legitimate line item, roughly half the budget.
Now read the budget critically — this paragraph is the bridge to Part 2. Each shard suffers, say, four crashes/year at ~7 minutes (28 min), monthly maintenance at 20 minutes (4 h), plus one bad incident — a corrupted snapshot, a confused runbook, a 90-minute datacenter network event. Total: ~5–6 hours. Fits in 8.8. But every line item leaned on the same three crutches: a human in the loop, a maintenance window, and exactly one copy of the truth. Four nines (53 min/year) forbids all three at once. That is Part 2's job description.
A snapshot contains a crossed book or the replay stream has a gap. Fast reopening would preserve or amplify corruption. An invariant or continuity failure must keep the venue closed and escalate to an older snapshot or repaired log.
Three nines = one truth copy + restart-and-replay + humans deciding. Snapshot cadence buys recovery speed; invariant checks catch silent corruption. The tier's crutches — humans, windows, single copies — are exactly what the next nine removes.