Overlay Networks

In one sentence: An overlay network spans multiple hosts by encapsulating container packets inside host-to-host UDP packets (VXLAN), guided by a shared control plane that knows which host each container lives on.

A network wearing a network

Container A (10.0.9.2, host 1) wants to reach container B (10.0.9.3, host 2). The physical network between the hosts has no idea what 10.0.9.x means — those addresses exist only inside the overlay. So each host runs a VXLAN tunnel endpoint: it takes A's entire packet, wraps it as the payload of a normal UDP packet addressed host-1 → host-2, and sends it over the real network. Host 2 unwraps and delivers the inner packet to B's sandbox. Both containers believe they share a switch; the "switch" is a tunnel. The missing ingredient is knowledge: host 1 must know B lives on host 2. That mapping — container IP → owning host — is the control plane, a small distributed database every member host keeps synchronized (via a shared store or gossip). Data plane wraps packets; control plane knows where to send them.

flowchart LR
  subgraph H1["host 1 (198.51.100.1)"]
    A["container A
10.0.9.2"]
    V1["VXLAN endpoint"]
    A --- V1
  end
  subgraph H2["host 2 (198.51.100.2)"]
    B["container B
10.0.9.3"]
    V2["VXLAN endpoint"]
    B --- V2
  end
  V1 == "UDP tunnel over the
physical network" ==> V2
  CP[("control plane
10.0.9.3 → host 2
10.0.9.2 → host 1")]
  CP -.-> V1
  CP -.-> V2
    

sequenceDiagram
  participant A as container A (10.0.9.2)
  participant V1 as VXLAN @ host 1
  participant NET as physical network
  participant V2 as VXLAN @ host 2
  participant B as container B (10.0.9.3)
  A->>V1: packet dst 10.0.9.3
  V1->>V1: control plane: 10.0.9.3 lives on host 2
  V1->>NET: UDP 198.51.100.1 → 198.51.100.2 [inner packet inside]
  NET->>V2: ordinary UDP delivery
  V2->>V2: unwrap — recover inner packet
  V2->>B: deliver to B's sandbox
  note over A,B: A and B never learn
they're on different machines