MiniDock Build 28: Connect Workloads Across Hosts

Overlay Networks

Design claim: A multi-host container network separates a control plane that knows endpoint location from a data plane that encapsulates packets.

The underlay cannot route what it does not know

Containers on different hosts must communicate with overlay addresses even though the physical network routes only host addresses. Connectivity is a set of scoped attachments, routes, and names, not an intrinsic property of a container record.

The tempting shortcut is straightforward: extend one host bridge across machines or expect the physical network to learn every container address. The shortcut works only while addresses are static and every participant shares one host or trust zone. Bridges are host-local and the underlay does not know overlay address ownership or container movement. Endpoint churn, isolation, or a second host exposes the missing scope and ownership information.

Separate location knowledge from packet movement

A distributed control plane maps overlay endpoints to hosts, and VXLAN endpoints encapsulate inner container packets inside routable host packets. The mechanism separates membership from packet movement so drivers can implement the same connection contract at different boundaries.

Read the topology by following an endpoint from its sandbox through the selected network boundary, keeping control information separate from packet flow.

  flowchart LR
    subgraph H1["host 1 (198.51.100.1)"]
      A["container A
10.0.9.2"]
      V1["VXLAN endpoint"]
      A --- V1
    end
    subgraph H2["host 2 (198.51.100.2)"]
      B["container B
10.0.9.3"]
      V2["VXLAN endpoint"]
      B --- V2
    end
    V1 == "UDP tunnel over the
physical network" ==> V2
    CP[("control plane
10.0.9.3 → host 2
10.0.9.2 → host 1")]
    CP -.-> V1
    CP -.-> V2
      

It places VXLAN endpoints on two hosts under a shared location map. Its central claim is that data-plane forwarding follows current control-plane ownership while preserving the container-visible packet; the labels therefore describe authority rather than decorative grouping.

Carry one network inside another

Container A (10.0.9.2, host 1) wants to reach container B (10.0.9.3, host 2). The physical network between the hosts has no idea what 10.0.9.x means — those addresses exist only inside the overlay. So each host runs a VXLAN tunnel endpoint: it takes A's entire packet, wraps it as the payload of a normal UDP packet addressed host-1 → host-2, and sends it over the real network. Host 2 unwraps and delivers the inner packet to B's sandbox. Both containers believe they share a switch; the "switch" is a tunnel. The missing ingredient is knowledge: host 1 must know B lives on host 2. That mapping — container IP → owning host — is the control plane, a small distributed database every member host keeps synchronized (via a shared store or gossip). Data plane wraps packets; control plane knows where to send them.

The source host looks up the destination location, wraps the unchanged inner packet, sends it through the underlay, and the destination host unwraps and delivers it. Trace both the forward and cleanup paths; connectivity is correct only when allocation and removal agree about ownership.

The second figure tests the same model in motion: it traces lookup, encapsulation, underlay delivery, decapsulation, and sandbox delivery.

  sequenceDiagram
    participant A as container A (10.0.9.2)
    participant V1 as VXLAN @ host 1
    participant NET as physical network
    participant V2 as VXLAN @ host 2
    participant B as container B (10.0.9.3)
    A->>V1: packet dst 10.0.9.3
    V1->>V1: control plane: 10.0.9.3 lives on host 2
    V1->>NET: UDP 198.51.100.1 → 198.51.100.2 [inner packet inside]
    NET->>V2: ordinary UDP delivery
    V2->>V2: unwrap — recover inner packet
    V2->>B: deliver to B's sandbox
    note over A,B: A and B never learn
they're on different machines
      

It traces lookup, encapsulation, underlay delivery, decapsulation, and sandbox delivery. The ordering is valid only when it continues to preserve the stated invariant under retries and interruption.

A stale map defeats a perfect tunnel

The location map is stale after a container moves, so a correctly encapsulated packet reaches the wrong host and must be dropped until control state converges. A stale or conflicting attachment must be refused or withdrawn locally rather than corrupting unrelated network state.

Data-plane forwarding follows current control-plane ownership while preserving the container-visible packet. The rule binds reachability to explicit, current membership within the correct scope.

Extend MiniDock without changing its endpoints

MiniDock can span hosts by adding location knowledge and encapsulation without changing the container network model. MiniDock now needs that connectivity property without being told which component to edit.