fivenines
13/38

Guided Problem

YourRide Build 11: Survive a Zone Loss

Time
20m
Level
foundation
Artifacts
not specified
Progress0%
Lesson 1.11 99.9%

Infrastructure baseline: one region, three zones

From 0.3: 99.9% = humans can react. The next decision is which failures machines absorb (a zone, an instance) and which humans handle (a database primary, the whole region). The services of 1.1 all live inside this picture.

The obvious design is to run several instances in one region and call the system highly available because individual processes are redundant. It makes the happy path simple, but it protects the wrong property. Loss of one zone must preserve sufficient service capacity and every acknowledged ride write, while whole-region loss remains an explicit boundary.

The rules of the baseline: every stateless service runs in at least three instances spread across three availability zones behind load balancing, on a container orchestrator with autoscaling — an instance or an entire zone can vanish and traffic re-routes in seconds without a human. Stateful systems get redundancy but manual judgment: PostgreSQL runs a primary with a synchronous standby in a second zone plus read replicas; failover is one command executed by on-call after a human confirms the primary is really dead (~5–10 minutes — affordable inside a 43-minute monthly budget, and it avoids the false-positive flapping that burns teams that automate failover before they can observe well). Redis runs replicated with the same policy; a queue and object store are managed, multi-zone services we buy.

flowchart TB
  DNS["DNS + CDN — static assets, TLS edge"] --> LB["Load balancer — cross-zone"]
  subgraph REGION ["Region: one, e.g. us-east"]
    subgraph AZA ["Zone A"]
      GA["Gateway + services ×N"]
      PGP[("PostgreSQL primary")]
      RDA[("Redis primary")]
    end
    subgraph AZB ["Zone B"]
      GB["Gateway + services ×N"]
      PGS[("PG sync standby")]
      RDB[("Redis replica")]
    end
    subgraph AZC ["Zone C"]
      GC["Gateway + services ×N"]
      PGR[("PG read replicas")]
    end
    LB --> GA
    LB --> GB
    LB --> GC
    PGP -. "sync replication" .-> PGS
    PGP -. "async" .-> PGR
    RDA -. "replication" .-> RDB
  end
  Q[["Managed queue — multi-zone"]]
  S3[("Object store — multi-zone")]
  REGION --> Q
  REGION --> S3
The 99.9% physical topology. Zone loss: automatic. Primary loss: human-driven failover, minutes. Region loss: an outage we accept at this tier — and the first thing Part 2 fixes.

Deployment v1 is a straight pipeline: merge → CI (tests, image build) → staging → production rolling update, zone by zone, with health checks gating each step and one-command rollback to the previous image. Infrastructure itself is code (Terraform or equivalent) from day one — not for elegance, but because Part 2's automation and Part 3's multi-region stamping are impossible over hand-built snowflakes.

The most common honesty failure at this tier: calling the system "multi-AZ" while a single-zone Redis, a cron box, or a pet bastion host quietly sits in the critical path. Do the audit: for each box in the diagram above, ask "zone A burns down — what exactly happens?" Write the answer next to the box.

Next step

See what actually stuck.

Take the practice scenarios now.