MiniDock Build 16: Restart Crashes Without Reviving Manual Stops

Supervision and Restart Policies

Design claim: Restart policy is a stateful decision over exit cause and operator intent, not a reflex applied to every stopped process.

Exit is evidence, not a decision

A crashed route worker should recover automatically, while a workload deliberately stopped for maintenance must remain stopped. Runtime behavior unfolds over time, so state and ownership must remain valid after the initiating request has returned.

The tempting shortcut is straightforward: restart whenever the main process is not running. The shortcut confuses one command or connection with the longer-lived process and resources it happens to touch. Crashes, clean completion, manual stops, and daemon restarts can all produce a non-running process but imply different operator intent. Asynchronous exit, retry, disconnect, or timeout exposes that mismatch immediately.

Combine policy with remembered intent

A restart policy is a promise made at create time. no: death is final (default — right for one-shot jobs). on-failure: resurrect only if the exit code is non-zero, optionally up to N tries — a zero exit means "finished successfully, let me rest." always: resurrect unconditionally, even after a clean exit, and also on daemon boot. unless-stopped: like always, with one human-respecting exception — if a user explicitly stopped it, daemon reboots leave it down. The distinction that matters: policies distinguish why the container is down (crashed vs. finished vs. told to stop).

A supervisor records policy, exit outcome, retry history, and manual-stop state before deciding whether and when to launch again. The mechanism records the durable fact separately from transient control and lets events update the model when reality changes.

Read the static view as custody and the dynamic view as evidence crossing that custody boundary.

  flowchart TB
    EV["exit event (code, was-user-stop?)"] --> Q0{"user ran stop?"}
    Q0 -- yes --> DOWN["stay Exited
(all policies respect stop)"]
    Q0 -- no --> QP{"policy?"}
    QP -- "no" --> DOWN
    QP -- "on-failure" --> QC{"exit code ≠ 0
and retries left?"}
    QC -- yes --> BK["wait backoff delay"]
    QC -- no --> DOWN
    QP -- "always /
unless-stopped" --> BK
    BK --> RS["restart container"]
    RS -. "crashes again?
delay doubles: 100ms, 200ms…" .-> EV
    style RS fill:#f0faf2,stroke:#1f7a34
      

It separates exit events, policy evaluation, backoff, and manual-stop state. Its central claim is that automatic recovery never overrides a newer explicit stop decision; the labels therefore describe authority rather than decorative grouping.

Classify before retrying

Without backoff, a container that crashes at startup becomes a fork bomb with extra steps — create, crash, create, crash, saturating CPU and flooding logs. Doubling the delay after each rapid failure (resetting once the container stays up briefly) turns a crash loop into a gentle heartbeat:

The shim reports an exit, the supervisor classifies it, applies policy and backoff, records the decision, and requests a restart only when allowed. The sequence matters because lifecycle decisions made from stale evidence become illegal or destructive operations.

The second figure tests the same model in motion: it contrasts crash recovery with a stop that suppresses later restart.

  sequenceDiagram
    participant S as Shim
    participant E as Engine
    participant C as Container
    S->>E: exit event (code 1) — uptime 2s
    E->>E: policy always → backoff 100ms
    E->>C: restart
    S->>E: exit event (code 1) — uptime 2s
    E->>E: backoff doubles → 200ms … then 400, 800 …
    E->>C: restart
    note over E: status shows "Restarting"—
backoff resets after a healthy stretch of uptime
      

It contrasts crash recovery with a stop that suppresses later restart. The ordering is valid only when it continues to preserve the stated invariant under retries and interruption.

Manual stop closes the restart loop

An operator stops a crash-looping worker, so queued backoff and future daemon recovery must not revive it. The failure case is authoritative input to the state model, not an exception to be hidden behind a successful command response.

Automatic recovery never overrides a newer explicit stop decision. The reusable rule keeps process reality, operator intent, and retained resources from collapsing into one status flag.

Give MiniDock a respectful supervisor

MiniDock adds supervision above lifecycle events while keeping user intent as durable state. MiniDock must preserve the lifecycle guarantee while leaving the current integration move to the learner.