Streaming data platform and real-time BI
- Evolve the nightly warehouse into a streaming lakehouse fed by CDC and the event backbone.
- Serve real-time surge, live ops dashboards, and ML features from one governed pipeline — without touching T0 paths.
From 1.13: yesterday's data today, and city ops feels it. From 2.4: every business fact is now already an event in the log — the analytics tap exists. From 1.5: surge runs on a 30 s batch loop; streaming makes it continuous. From 2.1: all of this is T3 — it must fail without touching the exchange.
Two feeds converge on the platform: the event backbone (trips, offers, payments — business facts, already flowing) and CDC (change-data-capture tailing database replication logs for entities that mutate without events — profiles, rate cards). Stream processors (Flink-class) consume both into a lakehouse in three refinement layers: bronze (raw, immutable, replayable), silver (cleaned, deduplicated by the same event ids from 2.4, conformed), gold (business aggregates: trips per city-minute, marketplace balance, driver utilization). The nightly warehouse (1.13) doesn't die — finance and regulators want stable daily truth; it becomes a consumer of silver, and the old quality gates guard gold.
Three consumers justify the platform: surge v2 — a streaming job joins the demand stream with supply per cell continuously; pricing still reads a cache (1.5's shape survives; only the freshness improved, seconds instead of 30), and if the stream stalls, surge holds last-known values — degraded, never absent (2.6's doctrine). Live ops — city teams watch marketplace balance and cancellation spikes minutes after reality, and incident commanders (2.9) get business impact in real time. ML features — a feature store computes online features (driver acceptance rate, rider cancel propensity) for the fraud models of 2.11 and future ranking. One pipeline, governed once: schema registry on every topic, lineage tracked, PII tagged at ingestion (2.11 depends on this).
flowchart LR
K[["Event backbone — trips, offers, payments (2.4)"]] --> FL["Stream processors"]
CDC[["CDC — DB replication logs"]] --> FL
PINGS[["Ping firehose (1.4)"]] --> FL
FL --> BR[("Bronze — raw, replayable")]
BR --> SV[("Silver — deduped, conformed")]
SV --> GD[("Gold — business aggregates")]
SV --> WH[("Nightly warehouse — finance, regulatory (1.13)")]
FL --> SC[("Surge cache v2 — seconds fresh")]
SC --> PRICE["Pricing (1.5) — unchanged interface"]
GD --> OPS["📊 Live city-ops dashboards"]
GD --> FS[("Feature store")]
FS --> FRAUD["Fraud scoring (2.11)"]
Late and out-of-order events are the permanent weather of streaming: a trip completed at 23:59 arriving at 00:03 belongs to yesterday. Use event time with watermarks, not processing time, or every window aggregate is subtly wrong — and city-ops dashboards that disagree with the finance warehouse destroy trust in both.