Degrade, don't die: overload protection
Why the busiest days break exchanges that redundancy can't save, and the priority scheme that keeps the system safe when demand exceeds capacity.
Every mechanism so far defends against failure. But exchanges historically die on their best days: panic selloffs, meme-stock manias — 10× normal volume arriving precisely when being up matters most. No replica helps when every replica is drowning; unbounded queues just convert overload into latency, then into memory exhaustion, then into the very crash you replicated against. Overload is an availability threat that redundancy cannot touch.
The defense is admission control with explicit priorities — decide at the door what matters when not everything fits:
flowchart TB
IN["Demand spike — 10× normal"] --> T1["Per-participant rate limits tighten first — heaviest senders throttled, fairness preserved"]
T1 --> Q{"Core queues still growing?"}
Q -- no --> N["Normal service"]
Q -- yes --> P["Shed by priority, lowest first"]
P --> P3["✂ delayed feeds, analytics, bulk queries"]
P --> P2["✂ order MODIFIES — clients cancel + re-enter instead"]
P --> P1["✂ NEW orders — rejected with explicit 'busy', client retries"]
P --> KEEP["NEVER shed: cancels, fills, risk checks, L1 market data"]
style KEEP stroke-width:3px
The protected set deserves its own justification — it's a risk ordering, not an importance ranking:
- Cancels are sacred. A trader who cannot cancel is trapped in positions during the exact volatility that caused the overload. Venues that dropped cancels under load turned bad days into lawsuits. Cancels also reduce book state — shedding them is self-defeating.
- Fills and risk checks are the integrity of the market itself — degrading them isn't degradation, it's corruption.
- L1 market data keeps every participant able to see the price. Depth (L2/L3) can conflate harder (1.8); the top of book must flow.
- New orders go before modifies because a modify is expressible as cancel + new, and under stress you want fewer verbs with clearer semantics.
Two disciplines make the ladder real. Bounded queues everywhere — every unbounded queue is a landmine that converts overload into an out-of-memory crash; when a bounded queue fills, the ladder fires, visibly and by design. And backpressure that stays inside the system: gateways learn core-queue depth and reject at the edge, spending zero core capacity on work that would be shed anyway. Note the rhyme with 1.6's message throttles and circuit breakers — the market-level and system-level safety mechanisms are the same idea at two altitudes: when demand exceeds safe capacity, degrade predictably and publicly, never silently.
- Overload is the failure mode redundancy can't fix — admission control is the answer.
- Shed by explicit priority; cancels, fills, risk, and L1 are never shed.
- Bounded queues + edge backpressure turn overload from a crash into a policy.