Streaming data platform and real-time BI
From 1.13: yesterday's data today, and city ops feels it. From 2.4: every business fact is now already an event in the log — the analytics tap exists. From 1.5: surge runs on a 30 s batch loop; streaming makes it continuous. From 2.1: all of this is T3 — it must fail without touching the exchange.
A first pass might let pricing query the streaming processor directly so every quote uses the freshest aggregate. That works only while the failure case is absent. Streaming outputs may become stale or replay, but lag and processor failure must not block or corrupt critical ride paths.
Two feeds converge on the platform: the event backbone (trips, offers, payments — business facts, already flowing) and CDC (change-data-capture tailing database replication logs for entities that mutate without events — profiles, rate cards). Stream processors (Flink-class) consume both into a lakehouse in three refinement layers: bronze (raw, immutable, replayable), silver (cleaned, deduplicated by the same event ids from 2.4, conformed), gold (business aggregates: trips per city-minute, marketplace balance, driver utilization). The nightly warehouse (1.13) doesn't die — finance and regulators want stable daily truth; it becomes a consumer of silver, and the old quality gates guard gold.
Three consumers justify the platform: surge v2 — a streaming job joins the demand stream with supply per cell continuously; pricing still reads a cache (1.5's shape survives; only the freshness improved, seconds instead of 30), and if the stream stalls, surge holds last-known values — degraded, never absent (2.6's doctrine). Live ops — city teams watch marketplace balance and cancellation spikes minutes after reality, and incident commanders (2.9) get business impact in real time. ML features — a feature store computes online features (driver acceptance rate, rider cancel propensity) for the fraud models of 2.11 and future ranking. One pipeline, governed once: schema registry on every topic, lineage tracked, PII tagged at ingestion (2.11 depends on this).
flowchart LR
K[["Event backbone — trips, offers, payments (2.4)"]] --> FL["Stream processors"]
CDC[["CDC — DB replication logs"]] --> FL
PINGS[["Ping firehose (1.4)"]] --> FL
FL --> BR[("Bronze — raw, replayable")]
BR --> SV[("Silver — deduped, conformed")]
SV --> GD[("Gold — business aggregates")]
SV --> WH[("Nightly warehouse — finance, regulatory (1.13)")]
FL --> SC[("Surge cache v2 — seconds fresh")]
SC --> PRICE["Pricing (1.5) — unchanged interface"]
GD --> OPS["📊 Live city-ops dashboards"]
GD --> FS[("Feature store")]
FS --> FRAUD["Fraud scoring (2.11)"]
Late and out-of-order events are the permanent weather of streaming: a trip completed at 23:59 arriving at 00:03 belongs to yesterday. Use event time with watermarks, not processing time, or every window aggregate is subtly wrong — and city-ops dashboards that disagree with the finance warehouse destroy trust in both.