Services, Tasks, and Reconciliation

In one sentence: The orchestrator runs one loop forever — observe actual state, diff against desired state, act to close the gap — which makes crash recovery, scaling, and rolling updates the same mechanism instead of three features.

Service → tasks → containers

A service ("web, image web:5, replicas 3") is pure intent. The orchestrator expands it into tasks — one desired container-run each, assigned by the scheduler to specific nodes (weighing available resources and spread across nodes for failure tolerance). Each engine runs its assigned tasks as ordinary containers, exactly as in Part 4. Task state flows back up; the loop never stops comparing. Nothing "handles" a node failure — its tasks simply stop reporting, the diff shows 2 of 3 running, and the loop schedules a replacement wherever capacity exists. Recovery isn't a feature; it's the loop's next iteration.

flowchart TB
  DS[("desired state
web: replicas 3, image web:5")]
  OBS["observe
task reports from all nodes"]
  DIFF{"diff: actual
vs desired?"}
  ACT["act
create / move / remove tasks"]
  SCHED["scheduler
picks nodes (resources + spread)"]
  W["workers run/stop containers"]
  DS --> DIFF
  OBS --> DIFF
  DIFF -- "gap found" --> ACT
  DIFF -- "converged" --> OBS
  ACT --> SCHED --> W
  W -- "status reports" --> OBS
    

Rolling updates: desired state with a schedule

Updating a service's image doesn't trigger special machinery — it edits desired state task-by-task. With update: one at a time, wait for healthy, the loop replaces one task, waits for its health gate (Lesson 24), then proceeds; a failure can pause or auto-rollback the rollout, leaving the other replicas serving:

sequenceDiagram
  participant O as Orchestrator loop
  participant S as Scheduler
  participant W2 as Worker 2
  participant W3 as Worker 3
  note over O: node 1 dies — its task stops reporting
  O->>O: diff: web has 2/3 replicas
  O->>S: need 1 replacement task
  S->>W3: run web task (had spare capacity)
  W3-->>O: running — converged again
  note over O: user: update web to image web:6 (one at a time)
  O->>W2: stop one web:5 task
  O->>W2: start web:6 task
  W2-->>O: Healthy ✓
  O->>W3: proceed to next replica …