MiniDock Build 03: Protect Dispatch from Batch Starvation

Cgroups: The Meters

Design claim: Visibility isolation must be paired with hierarchical accounting and enforcement or one container can exhaust the shared host.

Private views do not prevent starvation

Latency-sensitive gateway traffic and batch route computation must coexist without batch work consuming every CPU cycle or byte of memory. Begin with the guarantee visible to a workload or operator. Kernel machinery matters only after a simpler arrangement can no longer supply that guarantee.

The tempting shortcut is straightforward: assume namespaces also divide resource capacity. The shortcut is plausible on a quiet development host, where its missing boundary has not yet been tested. Namespaces change what a process can see, not how much shared CPU, memory, I/O, or process capacity it may consume. The constraint reveals which responsibility must move before the design can survive isolation or process failure.

Put policy where the kernel accounts

Cgroups form a tree. Each node carries resource budgets — memory ceiling, CPU weight or quota, I/O bandwidth, max PIDs — and every process belongs to exactly one node. Children can never exceed their parent's budget, which gives you delegation for free: give all containers a shared parent with 8 GB, and no misbehaving child can starve the host even if its own limit is generous. The engine's job is bookkeeping only — it creates a node per container and writes the limits; the kernel enforces them on every allocation and every scheduler tick.

Control groups place processes in hierarchical budgets, account for consumption, and enforce controller-specific limits. The named mechanism is the consequence of that separation: one layer owns policy, another enforces it, and callers receive a stable promise.

Read the topology as a map of authority. Durable knowledge, transient setup, and kernel enforcement should not blur into one box.

  flowchart TB
    ROOT["root cgroup
(all host resources)"]
    SYS["system.slice
sshd, journald …"]
    MD["minidock.slice
budget: 8 GB, 6 CPU"]
    C1["container A
mem 2 GB · cpu 1.0"]
    C2["container B
mem 4 GB · cpu 2.0"]
    C3["container C
mem 1 GB · cpu 0.5"]
    ROOT --> SYS
    ROOT --> MD
    MD --> C1
    MD --> C2
    MD --> C3
      

It shows workload groups nested under a host budget with independent controller limits. Its central claim is that every constrained process is accounted to exactly the resource boundary whose policy should contain it; the labels therefore describe authority rather than decorative grouping.

Membership makes limits real

Limits differ in personality. CPU limits throttle: exceed your quota and the scheduler simply stops running you until the next period — the app gets slow, not dead. Memory limits kill: when a cgroup's usage hits its ceiling and nothing can be reclaimed, the kernel's OOM killer terminates a process inside that cgroup (and only that cgroup — the blast radius is the budget node, never the host).

The engine creates a workload cgroup, writes its resource settings, moves the process into it, and lets the kernel schedule and account against that hierarchy. Trace the order carefully; each step establishes a fact the next step is entitled to use.

The second figure tests the same model in motion: it traces placement, accounting, throttling, and an out-of-memory outcome.

  sequenceDiagram
    participant E as Engine
    participant K as Kernel (cgroups)
    participant C as Container B
    E->>K: create node, write mem.max = 4 GB
    E->>K: place container B's process in node
    C->>K: allocate… allocate… (usage 3.9 GB)
    C->>K: allocate 200 MB
    K->>K: over ceiling, nothing reclaimable
    K->>C: OOM-kill process inside this cgroup only
    K-->>E: event: container B OOM-killed
    note over E: engine records exit reason
(Lesson 21 decides whether to restart)
      

It traces placement, accounting, throttling, and an out-of-memory outcome. The ordering is valid only when it continues to preserve the stated invariant under retries and interruption.

Contain exhaustion without hiding it

A route worker exceeds its memory ceiling while the gateway remains healthy, so the kernel must contain the failure to the constrained group. A sound boundary either preserves its promise or refuses the operation where the promise becomes impossible.

Every constrained process is accounted to exactly the resource boundary whose policy should contain it. This rule survives changes in implementation names because it describes ownership rather than a particular process tree.

Add budgets to MiniDock's walls

MiniDock adds meters beside its namespace walls so isolation covers both visibility and consumption. MiniDock needs the property now, but its learner must still decide where the responsibility belongs.