MiniDock Build 32: Reconcile Four Workers After Host Loss

Services, Tasks, and Reconciliation

Design claim: A service controller should continuously reconcile desired replicas with observed tasks rather than launch a fixed batch once.

Deployment is an ongoing comparison

Four route workers must remain available as containers exit and hosts disappear. Orchestration exists because actual state keeps changing after an initial deployment succeeds. Desired state must remain the durable reference point.

The tempting shortcut is straightforward: create four containers at deployment time and declare the service complete. The shortcut records actions, not the condition the system is supposed to maintain. Actual state changes after deployment, and a one-time launcher has no owner responsible for repairing replica count or replacing failed placements. Retries, readiness races, or host loss then produce duplicate work or unowned deficits.

Represent intent as services and attempts as tasks

A service ("web, image web:5, replicas 3") is pure intent. The orchestrator expands it into tasks — one desired container-run each, assigned by the scheduler to specific nodes (weighing available resources and spread across nodes for failure tolerance). Each engine runs its assigned tasks as ordinary containers, exactly as in Part 4. Task state flows back up; the loop never stops comparing. Nothing "handles" a node failure — its tasks simply stop reporting, the diff shows 2 of 3 running, and the loop schedules a replacement wherever capacity exists. Recovery isn't a feature; it's the loop's next iteration.

A reconciler compares the service specification with observed tasks, creates or removes tasks, and repeats after every relevant event. The mechanism represents intent explicitly, derives work from the current difference, and limits authority to the component able to commit that intent.

Read the control path from durable intent to observed tasks, distinguishing authority from execution.

  flowchart TB
    DS[("desired state
web: replicas 3, image web:5")]
    OBS["observe
task reports from all nodes"]
    DIFF{"diff: actual
vs desired?"}
    ACT["act
create / move / remove tasks"]
    SCHED["scheduler
picks nodes (resources + spread)"]
    W["workers run/stop containers"]
    DS --> DIFF
    OBS --> DIFF
    DIFF -- "gap found" --> ACT
    DIFF -- "converged" --> OBS
    ACT --> SCHED --> W
    W -- "status reports" --> OBS
      

It separates service specification, task slots, scheduler, workers, and observed state. Its central claim is that reconciliation acts on the current difference between desired and observed state and is safe to repeat; the labels therefore describe authority rather than decorative grouping.

Observe, diff, act, repeat

Updating a service's image doesn't trigger special machinery — it edits desired state task-by-task. With update: one at a time, wait for healthy, the loop replaces one task, waits for its health gate (Lesson 24), then proceeds; a failure can pause or auto-rollback the rollout, leaving the other replicas serving:

Desired replicas produce task slots, the scheduler places pending tasks, workers report outcomes, and the controller creates replacements until observed state converges. Each action should reduce a measured difference and remain safe when observation causes the loop to run again.

The second figure tests the same model in motion: it traces host loss, a two-task deficit, replacement placement, and renewed convergence.

  sequenceDiagram
    participant O as Orchestrator loop
    participant S as Scheduler
    participant W2 as Worker 2
    participant W3 as Worker 3
    note over O: node 1 dies — its task stops reporting
    O->>O: diff: web has 2/3 replicas
    O->>S: need 1 replacement task
    S->>W3: run web task (had spare capacity)
    W3-->>O: running — converged again
    note over O: user: update web to image web:6 (one at a time)
    O->>W2: stop one web:5 task
    O->>W2: start web:6 task
    W2-->>O: Healthy ✓
    O->>W3: proceed to next replica …
      

It traces host loss, a two-task deficit, replacement placement, and renewed convergence. The ordering is valid only when it continues to preserve the stated invariant under retries and interruption.

Host loss changes the difference, not the goal

A host carrying two workers is lost, so the controller must create exactly two replacements on eligible hosts rather than duplicate all four. Failure changes observed state, not the desired result; recovery should therefore repair only the new difference.

Reconciliation acts on the current difference between desired and observed state and is safe to repeat. The rule combines explicit conditions, quorum authority, and repeatable reconciliation without making workers owners of truth.

Give MiniDock a continuous controller

MiniDock turns container lifecycle events into continuous service repair. MiniDock can compose its existing APIs into continuous control while the design still requires a placement choice.