fivenines
21/40
Lesson 21 · Runtime

Supervision and Restart Policies

In one sentence: The shim catches every container death and reports it upward, where a per-container restart policy — no, on-failure, always, unless-stopped — decides whether the engine resurrects it, with exponential backoff against crash loops.

You already know

  • From Lesson 3: the shim parents the container and survives daemon restarts.
  • From Lesson 18: Running → Exited is an event; exit codes travel with it.

Four promises about death

A restart policy is a promise made at create time. no: death is final (default — right for one-shot jobs). on-failure: resurrect only if the exit code is non-zero, optionally up to N tries — a zero exit means "finished successfully, let me rest." always: resurrect unconditionally, even after a clean exit, and also on daemon boot. unless-stopped: like always, with one human-respecting exception — if a user explicitly stopped it, daemon reboots leave it down. The distinction that matters: policies distinguish why the container is down (crashed vs. finished vs. told to stop).

Architecture — the decision on every exit event
flowchart TB
  EV["exit event (code, was-user-stop?)"] --> Q0{"user ran stop?"}
  Q0 -- yes --> DOWN["stay Exited
(all policies respect stop)"] Q0 -- no --> QP{"policy?"} QP -- "no" --> DOWN QP -- "on-failure" --> QC{"exit code ≠ 0
and retries left?"} QC -- yes --> BK["wait backoff delay"] QC -- no --> DOWN QP -- "always /
unless-stopped" --> BK BK --> RS["restart container"] RS -. "crashes again?
delay doubles: 100ms, 200ms…" .-> EV style RS fill:#f0faf2,stroke:#1f7a34

Backoff: kindness to the host

Without backoff, a container that crashes at startup becomes a fork bomb with extra steps — create, crash, create, crash, saturating CPU and flooding logs. Doubling the delay after each rapid failure (resetting once the container stays up briefly) turns a crash loop into a gentle heartbeat:

Sequence — a crash-looping container under always
sequenceDiagram
  participant S as Shim
  participant E as Engine
  participant C as Container
  S->>E: exit event (code 1) — uptime 2s
  E->>E: policy always → backoff 100ms
  E->>C: restart
  S->>E: exit event (code 1) — uptime 2s
  E->>E: backoff doubles → 200ms … then 400, 800 …
  E->>C: restart
  note over E: status shows "Restarting"—
backoff resets after a healthy stretch of uptime

Anchor these

  • Policies branch on why it's down: crash (code ≠ 0), success (0), or user stop.
  • User stop wins over every policy — humans outrank automation.
  • Exponential backoff turns crash loops from outages into log lines.
Next step

See what actually stuck.

Take the practice scenarios now.