Events, Logs, and Metrics

Events, Logs, and Metrics

Design claim: Events, logs, and metrics are different evidence streams because they answer different operational questions.

Different questions require different evidence

Operators need to reconstruct lifecycle changes, inspect application output, and measure behavior over time without confusing one signal for another. Production evidence and authority cross every subsystem, so the completed design must preserve meaning at each seam rather than rely on a privileged center.

The tempting shortcut is straightforward: send every observation into one untyped text stream. The shortcut collapses distinct signals or controls into one convenient component and hides which promise failed. Discrete state changes, ordered process output, and numeric time series require different schemas, retention, indexing, and consumers. Compromise or host loss shows why independent evidence, refusal, and recovery paths must remain visible.

Preserve semantics before centralizing

The engine emits structured events for transitions, preserves log records from process streams, and exports sampled or aggregated metrics. The mechanism composes narrow contracts: each subsystem owns its decision, exports evidence, and limits the authority granted across the seam.

Read the map end to end, but stop at every boundary to identify its owner, input, output, and refusal behavior.

  flowchart LR
    subgraph SRC["sources"]
      LC["lifecycle + health
(state changes)"]
      APP["app stdout/stderr
(via shim)"]
      CG["cgroup counters
(cpu, mem, io)"]
    end
    BUS["event bus"]
    LD["log driver (pluggable)"]
    MET["metrics endpoint"]
    LC --> BUS
    APP --> LD
    CG --> MET
    BUS --> S1["subscriber: CI waiting on a build"]
    BUS --> S2["subscriber: audit trail"]
    LD --> S3["log storage / search"]
    MET --> S4["dashboards + alerts"]
      

It separates event, log, and metric producers, stores, and consumers. Its central claim is that each signal preserves the semantics needed to answer its class of question before correlation combines them; the labels therefore describe authority rather than decorative grouping.

Correlate without collapsing

Events answer "what happened?" — create, start, die, OOM, health-flip: discrete, structured, low-volume facts on one daemon-wide bus that any client can subscribe to. You've been generating them since Lesson 18; here they get an audience. Logs answer "what did the app say?" — high-volume text owned by the app, routed per-container to a pluggable driver (local files, journald, a remote shipper). Metrics answer "what does it cost?" — numeric time-series the engine reads straight from each container's cgroup, since the kernel was already counting. Keeping the three separate keeps each simple: nobody greps logs to learn a container died — that's an event; nobody emits an event per request — that's the app's logs.

A container start creates an event, application output enters its log path, resource controllers expose counters, and correlation identifiers connect the evidence. A successful path is convincing only when the same ownership map explains interruption and recovery.

The second figure tests the same model in motion: it correlates a lifecycle event, process output, and resource measurements over time.

  sequenceDiagram
    participant C as container
    participant E as engine
    participant B as event bus
    participant L as log driver
    participant M as metrics
    C->>L: "FATAL: out of memory allocating buffer"
    C->>E: exit (OOM-killed — L5)
    E->>B: event: die (id, exit 137, oom=true)
    B-->>B: all subscribers notified at once
    M-->>M: memory series for c-42 flatlines at the ceiling
    note over B,M: the event says WHAT, the log says WHY,
the metric shows IT COMING — you need all three
      

It correlates a lifecycle event, process output, and resource measurements over time. The ordering is valid only when it continues to preserve the stated invariant under retries and interruption.

Silence in one stream is not system health

A process is running but latency rises and no lifecycle event occurs, so metrics and logs must reveal degradation without fabricating an event. The boundary test should activate several independent controls without turning any one of them into universal authority.

Each signal preserves the semantics needed to answer its class of question before correlation combines them. The final rule is compositional: system correctness comes from explicit local guarantees that continue to hold together.

Give MiniDock three honest signal paths

MiniDock exposes separate evidence contracts and lets operators correlate them at stable container and service identities. MiniDock is complete when the whole ParcelFlow path can be explained from those contracts.