fivenines
23/40

Guided Problem

MiniDock Build 18: Stop Gracefully Before Forcing Exit

Time
25m
Level
intermediate
Artifacts
not specified
Progress0%
Lesson 23 · Runtime

Stopping Gracefully

Design claim: Stopping is a bounded protocol: request cooperative exit, wait for a deadline, then force termination.

Starting model

  • You can support stored logs, live attachment, and diagnostics without transferring primary lifecycle ownership.
  • The useful vocabulary at this point is deliberately small: SIGTERM, bounded shutdown interval, SIGKILL, PID 1.

Politeness needs a deadline

A service needs time to drain work and flush state, but the platform cannot wait forever for an unresponsive process. Runtime behavior unfolds over time, so state and ownership must remain valid after the initiating request has returned.

The tempting shortcut is straightforward: always send an uncatchable kill immediately or wait indefinitely after a polite signal. The shortcut confuses one command or connection with the longer-lived process and resources it happens to touch. Immediate force loses application cleanup, while unbounded waiting prevents deployment, shutdown, and resource reclamation. Asynchronous exit, retry, disconnect, or timeout exposes that mismatch immediately.

Make stop a two-phase protocol

SIGTERM means "please finish up": close connections, flush buffers, release locks. SIGKILL means "you no longer exist" — unhandleable, uncleanable. The grace period between them is the app's contract with the platform: handle TERM within N seconds or lose the right to clean up. The engine picks N per workload — 10 s suits a web server; a database checkpointing to disk may need minutes. Exit codes tell the story afterward: 143 (128+15) means TERM was honored; 137 (128+9) means the deadline passed and KILL fired.

The engine sends the configured stop signal to the container's main process, waits a finite grace period, and escalates only if it remains alive. The mechanism records the durable fact separately from transient control and lets events update the model when reality changes.

Read the static view as custody and the dynamic view as evidence crossing that custody boundary.

Architecture — why a tiny init sits at PID 1
  flowchart TB
    E["engine: stop c-42"] --> SH["shim"]
    SH -- "SIGTERM" --> I["tiny init (PID 1)
forwards signals · reaps zombies"] I -- "forward TERM" --> APP["app (PID 2)
closes conns, flushes"] APP -. "orphaned children
reparent here" .-> I SH -. "if deadline passes:
SIGKILL (unblockable)" .-> I

It shows running, stopping, exited, and forced-exit outcomes. Its central claim is that graceful shutdown is attempted exactly once and completion is bounded by an explicit timeout; the labels therefore describe authority rather than decorative grouping.

Observe exit before escalating

Two kernel quirks make PID 1 special inside a namespace: signals with default handlers are ignored for it (an app that never installs a TERM handler simply won't die politely), and orphaned child processes reparent to it, becoming zombies unless someone reaps them. Shell-wrapper entrypoints hit both — the shell holds PID 1, doesn't forward signals, doesn't reap. The fix is a tiny init as PID 1 that forwards signals to the real app and reaps orphans.

Stop records intent, signals the process, observes exit through the shim, or reaches the deadline and sends a forceful signal before recording the final state. The sequence matters because lifecycle decisions made from stale evidence become illegal or destructive operations.

The second figure tests the same model in motion: it orders signal, wait, observed exit, deadline, and escalation.

Sequence — a graceful stop and a stubborn one
  sequenceDiagram
    participant E as Engine
    participant S as Shim
    participant P as PID 1
    E->>S: stop (grace: 10s)
    S->>P: SIGTERM
    alt app handles TERM in time
      P->>P: cleanup, exit(0)
      S-->>E: exited 143 — graceful
    else deadline passes
      S->>P: SIGKILL
      S-->>E: exited 137 — forced, no cleanup ran
    end
    E->>E: mark "user-stopped" (restart policies stand down — L21)
      

It orders signal, wait, observed exit, deadline, and escalation. The ordering is valid only when it continues to preserve the stated invariant under retries and interruption.

The timeout is part of the contract

A gateway ignores the graceful signal while the host is shutting down, so MiniDock must force exit after the promised deadline. The failure case is authoritative input to the state model, not an exception to be hidden behind a successful command response.

Graceful shutdown is attempted exactly once and completion is bounded by an explicit timeout. The reusable rule keeps process reality, operator intent, and retained resources from collapsing into one status flag.

Give MiniDock bounded grace

MiniDock can honor application cleanup without surrendering lifecycle control. MiniDock must preserve the lifecycle guarantee while leaving the current integration move to the learner.

What carries forward

  • Graceful shutdown is attempted exactly once and completion is bounded by an explicit timeout.
  • Prefer graceful shutdown while guaranteeing bounded stop completion and correct restart behavior.
  • The rejected shortcut remains a diagnostic: if the design starts depending on it again, the original constraint has probably been lost.
Next step

See what actually stuck.

Take the practice scenarios now.