Retry Storms: Exponential Backoff, Jitter, Retry Budgets and Circuit Breakers

When a dependency slows down, the rest of the system tends to respond by trying harder. Every client retries the request that timed out, the struggling service suddenly sees two or three times its normal load, far more in a layered architecture, and a brief slowdown turns into a long outage. This is a retry storm. Retrying is not the problem in itself. The problem is retrying without deciding when, how often, and at which layer.

The multiplier effect

Picture a three-hop call chain: a frontend calls an API, the API calls an internal service, the internal service calls the database. If every layer retries a failed request three times, a single failure at the bottom becomes 3 x 3 x 3 = 27 attempts at the top. If the database was slow because it was overloaded, the number of requests reaching it grows geometrically at exactly the moment it can handle the least.

The first rule follows directly: retry at exactly one layer. Usually that is the layer closest to the failure that also knows whether the operation is safe to repeat. Every other layer propagates the error. Setups where an HTTP client library, a service mesh proxy and the application code all retry at once are surprisingly common, because each one looked reasonable when it was switched on.

Three strategies, and where each belongs

Fixed interval. Wait the same amount of time after every failure. Fine for a one-off script with a single client, harmful anywhere else. When thousands of clients fail together, they retry together; the delay just moves the wave forward in time.

Exponential backoff. The wait doubles on each attempt: 100 ms, 200 ms, 400 ms. It reduces total load but does not desynchronize clients that failed at the same moment. The waves get sparser but stay sharp.

Exponential backoff with full jitter. The upper bound doubles on each attempt, and the actual sleep is a random value between zero and that bound. Clients spread out over time and the service sees a smooth stream instead of spikes. This should be your default. Adding a small random offset around the bound also works, but full jitter is both simpler and spreads load better.

import random
import time

def retry(call, attempts=4, base=0.1, cap=5.0, deadline=None):
    for attempt in range(attempts):
        try:
            return call()
        except RetryableError:
            if attempt == attempts - 1:
                raise
            sleep = random.uniform(0, min(cap, base * 2 ** attempt))
            if deadline is not None and time.monotonic() + sleep >= deadline:
                raise
            time.sleep(sleep)

RetryableError is a deliberate distinction: not every error deserves a retry. Connection refused, timeouts, 503 and 429 are candidates. 400, 401, 404 and validation errors are not; the same request will get the same answer and only add load. If a 429 or 503 comes with a Retry-After header, honor it instead of your own calculation.

Budget retries, do not just count them

"At most three attempts" bounds the behavior of one request, not of the system. When a service is fully down, every request is still tried three times and load still triples. What you actually need is a cap on retries as a fraction of total traffic.

A retry budget works like this: each successful request adds a small credit to a bucket, each retry spends one unit. When the bucket is empty, the client stops retrying and returns the error immediately. During healthy periods occasional transient errors are retried transparently; when the dependency fails outright, the extra load stays limited to a small percentage of traffic.

You may not need to build this yourself. gRPC clients support both a retry policy and this kind of throttling in the service config:

{
  "methodConfig": [{
    "name": [{ "service": "orders.OrderService" }],
    "timeout": "2s",
    "retryPolicy": {
      "maxAttempts": 3,
      "initialBackoff": "0.1s",
      "maxBackoff": "1s",
      "backoffMultiplier": 2,
      "retryableStatusCodes": ["UNAVAILABLE"]
    }
  }],
  "retryThrottling": {
    "maxTokens": 10,
    "tokenRatio": 0.1
  }
}

With retryThrottling, each failed call decrements the token count by one and each successful call increments it by tokenRatio; once the count drops to half of the maximum, retries stop. Proxies such as Envoy offer a retry budget expressed as a percentage under a cluster's circuit breaker thresholds. Whatever tool you use, the idea is the same: limit retries as a ratio of traffic, not as a fixed number per request.

Deadlines flow downward

Treat timeouts and retries separately and you get a classic bug: the user facing request gives up after 2 seconds, but a lower layer keeps retrying with its own 2 second timeout three times, working for 6 seconds on an answer nobody is waiting for.

The fix is to set a deadline at the top and pass it down with every call. Each layer knows how much time is left and does not start an attempt it cannot finish. gRPC propagates deadlines natively; with HTTP services you have to carry it yourself in a header or a request context. The deadline argument in the example above exists for exactly this reason: if the deadline passes during the sleep, there is no point sleeping.

There is a lower bound as well. The timeout of the retrying layer should sit a little above the upper percentiles of the callee's normal latency. A timeout tuned to the median cuts off a meaningful share of slow but healthy requests and turns them into retries, so you raise the load with your own hands.

Circuit breakers: stop trying altogether

A retry budget limits retries, not first attempts. If a dependency is unresponsive for minutes, the first attempt of every request still waits for its timeout while holding threads and connections. A circuit breaker takes the dependency out of the path.

It has three states. Closed: requests pass through and the error rate is tracked. When the error rate in a window crosses a threshold, the breaker opens and requests fail immediately without reaching the dependency. After a cool-down it goes half-open and lets a limited number of probe requests through; if they succeed it closes, otherwise it opens again.

The value of a breaker is failing fast: requests do not pile up, the caller stays healthy, and you can answer from a cache, a default value or a degraded feature. If you have not designed what to return while the circuit is open, the breaker only makes the failure faster. That is still a win, just not the whole win.

Pitfalls

Retrying non-idempotent operations. A timeout does not mean the request failed; the server may have done the work and the response got lost. Payments, order creation or sending email retried without an idempotency key produce duplicates. Every write that may be retried needs server-side recognition of a second request carrying the same key.

Endless redelivery in queue consumers. If a message that fails processing goes straight back to the queue, one poison message traps the consumer in a hot loop. Queues need backoff, an attempt limit, and a dead letter queue for messages that exceed it.

Thundering herd on recovery. If every breaker goes half-open at the same moment when the dependency comes back, the recovering service takes all the traffic in its first second and falls over again. Add jitter to the open duration too, and keep the number of half-open probes small.

Proxy and application retrying on top of each other. If the mesh or load balancer retries, the application should not, or the other way round. Write down which layer owns retries; otherwise someone will add a third layer "for resilience" a year from now.

Not measuring retries. Requests that succeed only on the second or third attempt are an early sign of degradation. Export the retry count as its own metric; if first-attempt success drops while overall success looks flat, trouble is coming.

When not to retry

On interactive, latency sensitive paths, such as a page request a user is staring at, a single attempt is sometimes the best behavior: failing fast and offering a retry button beats waiting several seconds in the background. If the cause is overload and the server says so with a 429 or 503, nothing helps except honoring Retry-After. And if the operation is not idempotent, leave retries off until an idempotency key is in place.

A practical starting point: retries at a single layer, two or three attempts with full jitter, a retry budget around ten percent of traffic, a deadline that flows down from the top, and a circuit breaker in front of critical dependencies. You do not have to build all of it on day one, but the order matters: first decide which layer retries, then add jitter, then add the budget.