MiniDock Build 19: Separate Process State from Service Health

Health Checks

Design claim: Process state and service health answer different questions and must drive different automation.

Alive is necessary, not sufficient

A process may be alive while unable to serve traffic, and temporary startup failure must not be confused with permanent application failure. Runtime behavior unfolds over time, so state and ownership must remain valid after the initiating request has returned.

The tempting shortcut is straightforward: use process existence as the readiness signal. The shortcut confuses one command or connection with the longer-lived process and resources it happens to touch. Deadlocks, exhausted dependencies, bad configuration, and warm-up can all leave a running process unable to satisfy its service contract. Asynchronous exit, retry, disconnect, or timeout exposes that mismatch immediately.

Probe the service contract separately

A deadlocked web server is Running forever and serving nothing. The image (or run config) therefore declares a probe — "GET /health returns 200" — plus four knobs: interval (how often), timeout (how long a probe may hang), retries (consecutive failures before declaring unhealthy — one blip shouldn't flip state), and start-period (a warm-up window where failures don't count, so slow-booting apps aren't condemned at birth). Health is a separate little state machine layered on Running:

A health probe produces a separate state with start period, interval, timeout, retries, and success criteria defined at the service boundary. The mechanism records the durable fact separately from transient control and lets events update the model when reality changes.

Read the static view as custody and the dynamic view as evidence crossing that custody boundary.

  stateDiagram-v2
    [*] --> Starting: container starts (start-period)
    Starting --> Healthy: first probe passes
    Starting --> Starting: failures ignored (warm-up)
    Healthy --> Healthy: probe passes
    Healthy --> Unhealthy: N consecutive failures
    Unhealthy --> Healthy: probe passes again
    note right of Unhealthy
      engine emits an event —
      consumers decide what to do
      (restart, de-route, alert)
    end note
      

It places health transitions beside, not inside, the process lifecycle. Its central claim is that health reports service behavior and never substitutes for the truth of whether the process exists; the labels therefore describe authority rather than decorative grouping.

Turn repeated evidence into health state

Mechanically each probe is a tiny exec (Lesson 22): run the command in the container's namespaces, exit 0 = pass. The engine only reports health transitions as events — acting on them (restarting, pulling from a load balancer, gating dependent startups in Lesson 35) belongs to consumers. Separating detection from reaction keeps the engine simple and the policy pluggable:

The engine schedules probes, ignores configured startup noise, records consecutive results, and changes health without rewriting the container lifecycle state. The sequence matters because lifecycle decisions made from stale evidence become illegal or destructive operations.

The second figure tests the same model in motion: it traces startup grace, probe results, retry thresholds, and recovery.

  sequenceDiagram
    participant E as Engine (timer)
    participant C as Container
    participant O as Event consumers
    loop every interval
      E->>C: exec probe: GET /health
      C-->>E: exit 0 — pass
    end
    note over C: app deadlocks (process still Running)
    E->>C: probe → timeout (counts as failure 1)
    E->>C: probe → failure 2
    E->>C: probe → failure 3 = retries
    E->>O: event: c-42 Healthy → Unhealthy
    O->>O: policy: restart it / de-route traffic
      

It traces startup grace, probe results, retry thresholds, and recovery. The ordering is valid only when it continues to preserve the stated invariant under retries and interruption.

Unhealthy does not mean exited

The route API keeps running but fails its dependency probe repeatedly, so traffic automation must react while lifecycle remains running. The failure case is authoritative input to the state model, not an exception to be hidden behind a successful command response.

Health reports service behavior and never substitutes for the truth of whether the process exists. The reusable rule keeps process reality, operator intent, and retained resources from collapsing into one status flag.

Give MiniDock two honest signals

MiniDock adds a parallel health state that consumers may use without corrupting lifecycle semantics. MiniDock must preserve the lifecycle guarantee while leaving the current integration move to the learner.