MiniDock Build 31: Keep Cluster Authority with a Quorum

Swarm Architecture

Design claim: Cluster authority needs a replicated manager quorum, while workers execute committed assignments without independently inventing cluster intent.

A cluster needs one truth after failure

A cluster must continue making one coherent scheduling decision after a manager failure without allowing competing leaders. Orchestration exists because actual state keeps changing after an initial deployment succeeds. Desired state must remain the durable reference point.

The tempting shortcut is straightforward: let every node mutate cluster state or rely on one unreplicated manager. The shortcut records actions, not the condition the system is supposed to maintain. Peer mutation creates conflicting truth, while a single manager is unavailable after failure and cannot safely distinguish isolation from ownership. Retries, readiness races, or host loss then produce duplicate work or unowned deficits.

Replicate authority, not every action

Managers replicate cluster intent through consensus and elect one leader; workers accept committed assignments and report observed state. The mechanism represents intent explicitly, derives work from an authorized decision, and limits authority to the component able to commit that intent.

Read the control path from durable intent to observed execution, distinguishing authority from work.

  flowchart TB
    subgraph MGRS["managers (the truth)"]
      M1["manager 1 — LEADER
orders all writes"]
      M2["manager 2 (follower)"]
      M3["manager 3 (follower)"]
      M1 -- "Raft replication" --- M2
      M1 -- "Raft replication" --- M3
    end
    ST[("replicated state store
cluster intent · nodes · assignments")]
    M1 --- ST
    W1["worker 1
runs assignments, reports status"]
    W2["worker 2
runs assignments, reports status"]
    M1 -- "assign work" --> W1
    M1 -- "assign work" --> W2
    U["user: deploy workload"] --> M1
      

It separates a manager quorum and leader from worker execution nodes. Its central claim is that only a manager with quorum-backed authority commits cluster state; the labels therefore describe authority rather than decorative grouping.

Commit before assigning work

Every node runs the same engine plus a cluster process, but roles differ. Managers hold the truth: cluster intent, node records, and work assignments live in a state store replicated across managers with the Raft consensus algorithm — one elected leader orders all writes, and a write commits only when a majority (quorum) has stored it. Three managers tolerate one loss; five tolerate two. Lose quorum and the cluster becomes read-only: running containers keep running, but nothing new can be decided — a deliberate choice of consistency over availability, because two half-clusters each scheduling their own copies would be worse. Workers hold no cluster truth: they receive committed assignments, run containers, and report status. Losing a worker loses only capacity.

A client submits to a manager, the leader commits the change through quorum, scheduling assigns work, and workers execute and report outcomes. Commit must precede execution so a minority partition cannot create a competing cluster history.

The second figure tests the same model in motion: it traces proposal, quorum commit, work assignment, leader loss, and re-election.

  sequenceDiagram
    participant U as User
    participant L as Leader
    participant F1 as Follower 2
    participant F2 as Follower 3
    U->>L: service create web (replicas 3)
    L->>F1: replicate log entry
    L->>F2: replicate log entry
    F1-->>L: stored
    note over L: 2 of 3 = majority → COMMIT
(no need to wait for F2)
    L-->>U: accepted — state is durable
    note over L,F2: if the leader dies now, a new leader
is elected from nodes holding the entry
      

It traces proposal, quorum commit, work assignment, leader loss, and re-election. The ordering is valid only when it continues to preserve the stated invariant under retries and interruption.

Minority isolation removes authority

The leader becomes isolated from a majority, so it must stop committing changes while the remaining quorum elects a new leader. Failure changes observed state, not the desired result; recovery should therefore repair only the new difference.

Only a manager with quorum-backed authority commits cluster state. The rule combines explicit conditions, quorum authority, and repeatable reconciliation without making workers owners of truth.

Put MiniDock's cluster truth behind quorum

MiniDock can scale control across hosts by separating replicated authority from replaceable execution. MiniDock can compose its existing APIs into continuous control while the design still requires a placement choice.