Reading the nines: availability as a budget
How to translate an availability percentage into a downtime budget, and why chaining components eats that budget alarmingly fast.
The requirements map contains paths with different user consequences. An availability percentage is meaningful only after we name the user-visible question it measures.
It is tempting to give every component the advertised availability target and assume the complete trading path inherits the same number. The constraint it misses is simple: Allocate one downtime budget to the complete user-visible path, compose serial dependencies explicitly, and count redundancy only when its failure modes are genuinely independent.
"99.9% available" is a contract expressed as a percentage of time the system works. Invert it and it becomes something you can design against: a downtime budget.
| Target | Downtime / year | Downtime / month | What recovery must look like |
|---|---|---|---|
| 99.9% (three nines) | 8 h 46 m | 43.8 min | A human can be woken up, diagnose, restart, and replay. A handful of 30–60 minute incidents a year fits. |
| 99.99% (four nines) | 52.6 min | 4.4 min | No time for humans in the loop. Failover must be automatic; maintenance can no longer mean downtime. |
| 99.999% (five nines) | 5.26 min | 26 s | Failover in seconds, invisible to most users. Whole-datacenter loss must be survivable. Change itself becomes the main risk. |
We build a 24/7-capable venue (think a global crypto-style exchange, or a traditional one that refuses to rely on overnight maintenance windows). That makes the math honest: there is no free "market closed" time to hide repairs in.
One more piece of arithmetic shapes everything that follows. Availability composes multiplicatively across components a request must pass through in series:
flowchart LR
A["Gateway 99.95%"] --> B["Risk 99.95%"] --> C["Matching 99.95%"] --> D["Market data 99.95%"]
D --> R["End to end: 0.9995 x 0.9995 x 0.9995 x 0.9995 = 99.80%"]
style R fill:none,stroke-dasharray: 5 5
The reverse is also true and is the engine of Parts 2 and 3: redundancy composes in your favor. Two independent 99.9% replicas fail together only if both are down at once — theoretically 99.9999%. The word doing the heavy lifting is independent: shared disks, shared bugs, and shared operators quietly re-correlate your failures. Much of this course is a war on hidden correlation.
Two 99.9% replicas in the same rack do not provide six nines when the rack loses power. The multiplication only works for independent failures; shared power, deployments, disks, clocks, and operators can turn two boxes back into one failure domain.
Each extra nine divides your downtime budget by ten — and changes who can perform recovery (human → machine). Serial hops multiply against you; independent replicas multiply for you. Hidden correlation is the enemy of redundancy.