Operating at three nines: crash, restart, replay

At three nines, the recovery plan for the core is honest and simple: one authoritative engine per shard; if it dies, restart it and replay. Walk the clock through an incident:

flowchart LR
    A["14:02:00 — engine process dies"] --> B["14:02:15 — health check fails, page fires"]
    B --> C["14:04 — on-call confirms, triggers restart runbook"]
    C --> D["14:05 — snapshot loaded (5 min old)"]
    D --> E["14:06 — journal replayed to head"]
    E --> F["14:07 — gateways reconnect, resume"]
    F --> G["~5–7 minutes downtime for this shard's symbols"]
    

Engineering that makes the timeline real:

• Snapshot cadence bounds replay time. Snapshot every 5 minutes; replay runs ~50× real time; worst-case replay ≈ 6 seconds. The journal and snapshots live on storage that survives the process — at this tier, a replicated disk is acceptable.
• Monitor the four golden signals per component — traffic, errors, latency, saturation — plus two exchange-specific ones: sequence-gap alarms (any consumer seeing gaps in the event stream) and invariant checks (reserved balances match live orders, ledger matches replay — from 1.9/1.10). Alert on symptoms users feel, not on every twitching metric.
• Maintenance uses the budget deliberately. Upgrades happen in scheduled low-activity windows: drain, upgrade, restart, replay, verify. Planned and boring — at this tier, downtime for change is a legitimate line item, roughly half the budget.

Now read the budget critically — this paragraph is the bridge to Part 2. Each shard suffers, say, four crashes/year at ~7 minutes (28 min), monthly maintenance at 20 minutes (4 h), plus one bad incident — a corrupted snapshot, a confused runbook, a 90-minute datacenter network event. Total: ~5–6 hours. Fits in 8.8. But every line item leaned on the same three crutches: a human in the loop, a maintenance window, and exactly one copy of the truth. Four nines (53 min/year) forbids all three at once. That is Part 2's job description.