Retries · System Design Concepts

A retry storm: the inventory service goes down under its own retries

Retries feel like the responsible thing to add. A call failed? Try again. It’s in almost every codebase:

for _ in range(3):
    try:
        return check_stock(sku)
    except TimeoutError:
        continue

It looks harmless, but this loop can turn a service that’s only slow into one that’s down. Here’s how that happens, and the three fixes that make retries safe.

The scenario

Say you built the order service. Your teammate built the inventory service. For every order, your service calls inventory to check the stock.

One day, inventory slows down for a few seconds. Maybe a slow database query or a long garbage collection pause. At that exact moment, 1,000 orders are calling it.

How a slow service becomes a dead one

Those 1,000 calls time out. Your retry loop fires immediately. So now inventory gets:

the 1,000 retries, plus
every new order that keeps arriving.

New orders plus instant retries push the load on inventory past its capacity

The load on inventory goes up, exactly when it can handle the least. More calls time out, more retries fire, and the load climbs again. Inventory was just slow. Now it’s down, and from the outside it looks like your teammate’s bug.

That’s a retry storm: retries that add load faster than a struggling service can recover. It’s like calling someone ten times because they didn’t pick up. They won’t pick up faster.

Fix 1: exponential backoff

Don’t retry immediately. Wait, and wait longer after every failure:

wait = base × 2^attempt     →  1s, 2s, 4s, 8s ...

Always cap the wait (for example, at 8 or 30 seconds) so it can’t grow forever.

Exponential backoff: wait 1s, then 2s, then 4s, so inventory gets time to recover

The gaps give inventory room to catch up. That’s how it recovers instead of being buried.

The catch: everyone backs off together

Backoff alone has a problem. All 1,000 orders timed out at the same moment. So they all wait exactly 1 second, and all retry at the same moment again. Then they all wait 2 seconds and hit together again. Backoff spaces out the waves, but every wave is still a spike.

Without jitter, all 1,000 retries hit inventory at the same moment, far above capacity

Fix 2: add jitter

Add randomness to the wait. The most common version is full jitter: wait a random time between zero and the backoff:

wait = random(0, min(cap, base × 2^attempt))

Now the 1,000 retries spread across the whole second instead of landing together.

With full jitter, the retries spread out and stay under capacity

You can see the difference with a quick simulation:

import random
from collections import Counter

CLIENTS = 1000      # orders that timed out at the same moment
BACKOFF = 1.0       # seconds before the first retry


def busiest_100ms(jitter: bool) -> int:
    """How many retries land in the busiest 100 ms slot."""
    slots = Counter()
    for _ in range(CLIENTS):
        wait = random.uniform(0, BACKOFF) if jitter else BACKOFF
        slots[int(wait * 10)] += 1
    return max(slots.values())


random.seed(42)
print("No jitter:  ", busiest_100ms(jitter=False), "retries in the busiest 100 ms")
print("Full jitter:", busiest_100ms(jitter=True), "retries in the busiest 100 ms")

No jitter:   1000 retries in the busiest 100 ms
Full jitter: 117 retries in the busiest 100 ms

Same number of retries, but the peak drops by about 8×. AWS studied this in detail (Exponential Backoff and Jitter), and found that jittered backoff does far less total work than backoff without jitter.

Fix 3: limit the attempts

Retrying forever just keeps the pressure on. Set a limit, for example 3 attempts, and then fail fast with a clear error:

503 Service Unavailable: inventory is unavailable, please try again later

A clear error lets the caller decide what to do: show a message, queue the order, or fall back. An endless retry loop hides the problem and holds connections open while it waits.

Putting it together

All three fixes fit in a few lines:

import random
import time


class TransientError(Exception):
    """A failure worth retrying: timeout, connection reset, 503, 429."""


def call_with_retries(call, max_attempts=3, base=1.0, cap=8.0):
    for attempt in range(max_attempts):
        try:
            return call()
        except TransientError:
            if attempt == max_attempts - 1:
                raise  # limit reached: fail fast with a clear error
            backoff = min(cap, base * 2 ** attempt)  # 1s, 2s, 4s, ... up to cap
            time.sleep(random.uniform(0, backoff))   # full jitter

In real Python projects you’d usually use a library. With tenacity, the same policy is a decorator. Its wait_random_exponential is full jitter:

from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_random_exponential


@retry(
    retry=retry_if_exception_type(TransientError),      # only retry failures that may go away
    wait=wait_random_exponential(multiplier=1, max=8),  # backoff + full jitter, capped at 8s
    stop=stop_after_attempt(3),                         # limit
    reraise=True,
)
def check_stock(sku: str) -> dict:
    ...

Only retry what’s safe to retry

Backoff, jitter and a limit control when you retry. Two more rules decide whether you should:

Retry only failures that might go away: timeouts, connection errors, 502, 503, 504, and 429 Too Many Requests. If the server sends a Retry-After header, respect it. Don’t retry 400, 401, 403 or 404; the same request will fail the same way.
Retry only calls that are safe to repeat. Checking stock is a read, so repeating it is harmless. Placing an order or charging a card isn’t. If the first attempt actually succeeded and only the response was lost, a retry does it twice. Make those calls idempotent first, usually with an idempotency key. I covered this in Idempotency.

Going further: retries multiply across layers

One more trap for bigger systems. If the API gateway retries 3 times, your order service retries 3 times, and its database client retries 3 times, a single user request can become up to 3 × 3 × 3 = 27 calls on the bottom service. The usual advice, from the Google SRE book, is to retry at one layer only, and to cap retries as a share of total traffic (a “retry budget”) instead of only per request.

Checklist

✅ Never retry immediately in a loop.
✅ Exponential backoff: wait longer after every failure, with a cap.
✅ Jitter: randomise the wait so clients don’t retry in sync.
✅ Limit attempts, then fail fast with a clear error.
✅ Retry only transient failures, and respect Retry-After.
✅ Retry only idempotent calls, or make them idempotent first.
✅ Retry at one layer, not at every layer.

Retries aren’t free. Backoff, jitter, limit, and they’re safe.