Chaos engineering and gamedays: rehearsing the theory

Chaos engineering is hypothesis testing against the running system: state the expected behavior ("killing a dispatch shard causes ≤ 20 s of elevated wait in its cells and zero SLI movement elsewhere"), inject the fault in production — the canary cell (3.3) exists exactly so this is safe — with a guardrail that halts the experiment on real SLO burn, and compare. Passed experiments become continuous: instance kills run daily unattended; zone loss, quorum-member loss, and control-plane blackholes run weekly; a full region re-home runs quarterly as a scheduled, announced event. The goal is a system for which nothing that happens on its worst day happens for the first time. This is also where cell isolation (3.3's pitfall) and static stability (3.5's pitfall) get their teeth: both listed a test — this lesson is where those tests live.

Gamedays rehearse the humans: a scenario master injects a compound failure (region loss during a control-plane outage — the compounds are where automation hands off to people), on-call runs the real process from 2.9 — commander, roles, mitigation-first — and the debrief audits the process: was detection as fast as claimed? did the runbook match reality? did the commander have the authority the evacuation criteria assume? Gamedays are to incident response what worked examples are to learning — the full solution path, walked under low stakes, so the schema exists before the 3 a.m. exam.

flowchart LR
  HYP["Hypothesis — expected behavior + blast radius"] --> INJ["Inject fault — canary cell first, guardrailed"]
  INJ --> OBS["Observe SLIs vs prediction"]
  OBS --> MATCH{"Reality = claim?"}
  MATCH -- "yes" --> AUTO["Automate — run continuously at increasing scope"]
  MATCH -- "no" --> FIX["Fix the gap — architecture or runbook"]
  FIX --> HYP
  AUTO --> CAT["Fault catalog: instance daily, zone weekly, region quarterly, compound in gamedays"]