fivenines
16/40
Lesson 16 · Distribution

Push and Dedup

In one sentence: A push inverts the pull — blobs first, manifest last — and asks "do you already have this?" before every upload, so shared base layers cross the wire exactly once, ever.

You already know

  • From Lesson 15: pull is manifest-first, and tagging last gives crash safety.
  • From Lesson 14: HEAD on a blob URL checks existence without transferring bytes.

Upload the index last

Push order is pull order reversed, for the same crash-safety reason: a manifest must never exist on the registry while blobs it references are missing — anyone pulling it would fail. So the engine uploads all blobs first, and the manifest only when every referenced digest is safely stored (registries enforce this: a manifest referencing a missing blob is rejected). Before each upload the engine sends a cheap HEAD: registries are content-addressed too, so if any image ever uploaded that blob — even to another repository — the bytes can be skipped, or linked across repos with a cross-repo mount request.

Architecture — the push decision, per blob
flowchart TB
  B["next blob in manifest"] --> H{"HEAD: registry
has this digest?"} H -- "yes (any repo)" --> SKIP["skip or cross-repo mount
(zero bytes transferred)"] H -- no --> UP["chunked upload
POST session + PUT chunks"] UP --> V{"registry re-hashes:
digest matches?"} V -- yes --> OK["blob stored"] V -- no --> REJ["rejected — retry"] SKIP --> DONE{"all blobs present?"} OK --> DONE DONE -- yes --> M["PUT manifest (last!)"] style M fill:#f0faf2,stroke:#1f7a34
Sequence — pushing an image whose base is already known
sequenceDiagram
  participant E as Engine
  participant R as Registry
  E->>R: HEAD blobs/sha256:aa (base layer)
  R-->>E: 200 — exists (someone pushed it before)
  E->>R: HEAD blobs/sha256:cc (app layer)
  R-->>E: 404 — missing
  E->>R: POST upload session for cc
  E->>R: PUT chunks … done (digest cc)
  R->>R: re-hash ✓ — stored
  E->>R: PUT manifest (references aa, cc)
  R->>R: verify all referenced blobs exist ✓
  R-->>E: 201 — image published
    

Anchor these

  • Blobs first, manifest last — the registry never serves a broken image.
  • HEAD-before-upload makes pushes incremental for free; popular bases upload once, globally.
  • The registry re-hashes uploads — clients can't poison a digest.
Next step

See what actually stuck.

Take the practice scenarios now.