> ## Content Index
> Fetch the complete content index at: https://stalin-pereira.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Implementing Exponential Backoff to Prevent Thundering Herds
- URL: https://stalin-pereira.com/implementing-exponential-backoff-to-prevent-thundering-herds/
- Published: 2025-12-28T10:51:00.000Z
- Updated: 2026-10-16T19:23:21.000Z
- Description: Immediate retries turn a brief blip into a self-inflicted outage. How exponential backoff with full jitter works, a production-ready retry helper in TypeScript, Envoy and gRPC configs, and where retries should and should not live.
- Author: Stalin Pereira
- Tags: Distributed Systems

Systems rarely fall over just because traffic grew. They fall over because of the *rhythm* of that traffic: thousands of clients doing the same thing at the same moment. A popular cache key expires and every request misses at once. A database fails over and every connection pool reconnects in the same second. A dependency returns a few 503s, every caller retries immediately, and a brief blip becomes an outage the struggling service caused for itself.

That pattern is the thundering herd. The usual first defence against it is exponential backoff with jitter. The idea is easy to state: wait longer after each failure, and randomise the wait so clients drift apart. Getting it right is harder. The formula matters, the retry cap matters, which errors you retry matters, and where in the stack the retries happen matters most of all.

This post covers the maths, a production-ready retry helper in TypeScript, how to configure the same behaviour in Envoy and gRPC, and the cases where backoff is not the answer.

## Why immediate retries make outages worse

When a request fails, retrying straight away looks harmless. One request, one more try. But look at it from the server's side. If the server is failing because it is overloaded, every client that retries immediately doubles the load it puts on that server, at exactly the moment the server can least absorb it. Fixed-interval retries ("try again every second") are only slightly better. The pressure never drops, because the retry rate stays constant no matter how long the outage lasts.

Retries also multiply across layers. If an edge service, an API, and a data-access layer each make up to 4 attempts, one user action can turn into 4 × 4 × 4 = 64 attempts against the database. The Google SRE book's chapter on [addressing cascading failures](https://sre.google/sre-book/addressing-cascading-failures/?ref=stalin-pereira.com) uses this example and gives three rules worth memorising: limit retries per request, don't retry at multiple levels, and always use randomised exponential backoff.

## Exponential backoff: the basic shape

Exponential backoff grows the wait after each failed attempt by a constant factor, usually 2:

```text
delay(attempt) = min(cap, base * 2^attempt)

base = 100ms, cap = 5s
attempt 0 -> 100ms
attempt 1 -> 200ms
attempt 2 -> 400ms
attempt 3 -> 800ms
attempt 4 -> 1.6s
...
attempt 6+ -> 5s (capped)
```

Three parameters do the work:

- **Base** sets how quickly you retry the first time. Too small and you hit the server again before anything has changed. Too large and a one-off network blip costs the user a visible delay.
- **Cap** stops the delay from growing without limit. Without it, attempt 10 at a 100ms base waits over 100 seconds, which is useless for an interactive request.
- **Max attempts** (or an overall deadline) decides when to give up. Backoff spreads retries out over time. It doesn't make an unlimited number of them safe.

## Backoff alone still produces waves

Pure exponential backoff has a flaw: it is deterministic. If 1,000 clients fail at 12:00:00.000, they all wait exactly 100ms and retry together, then wait 200ms and retry together again. You have spaced the spikes out, but they are still spikes. The herd is still a herd.

Jitter fixes this by randomising each delay so clients that failed together retry at different times. Marc Brooker's post on the AWS Architecture Blog, [Exponential Backoff and Jitter](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/?ref=stalin-pereira.com), is the standard reference here. It simulates contending clients and compares several jitter strategies.

### The jitter variants

| Strategy            | Formula                                         | Behaviour                                                              |
| ------------------- | ----------------------------------------------- | ---------------------------------------------------------------------- |
| No jitter           | min(cap, base \* 2^n)                           | Clients retry in synchronised waves.                                   |
| Additive jitter     | min(cap, base \* 2^n) + random(0, small)        | Only slightly better: the spread is tiny compared with the delay.      |
| Equal jitter        | t = min(cap, base \* 2^n); t/2 + random(0, t/2) | Keeps a guaranteed minimum wait; spreads over the upper half.          |
| Full jitter         | random(0, min(cap, base \* 2^n))                | Spreads retries across the whole window. A good default.               |
| Decorrelated jitter | min(cap, random(base, prev \* 3))               | Each delay depends on the previous one rather than the attempt number. |

A common hand-written formula is `2^attempt + random_milliseconds()`. That's additive jitter, and when the random part is small compared with the exponential part, clients stay almost as tightly bunched as they were with no jitter. In the AWS simulations, every jittered approach did far less work than no jitter, with full and decorrelated jitter coming out best. Full jitter is the easiest to reason about, so it's the one I'd reach for unless you have a reason not to.

Full jitter can choose a delay close to zero. That's fine: across many clients the average delay is half the window, and the point is to spread load, not to guarantee any single client a minimum wait. If you need a floor, for example to respect a rate limit, use equal jitter or honour a `Retry-After` header.

## A retry helper you can actually ship

Most retry bugs are not in the backoff maths. They come from retrying the wrong errors, ignoring the caller's deadline, or ignoring what the server said. The helper below handles all three. It runs on Node.js 18+ and modern browsers, using the built-in `fetch` and `AbortSignal.timeout`.

```typescript
type RetryOptions = {
  maxAttempts: number;   // total attempts, including the first
  baseMs: number;        // first backoff ceiling
  capMs: number;         // never sleep longer than this
  signal?: AbortSignal;  // caller's overall deadline
};

class RetryableError extends Error {
  readonly retryAfterMs?: number;
  constructor(message: string, retryAfterMs?: number) {
    super(message);
    this.retryAfterMs = retryAfterMs;
  }
}

function fullJitter(attempt: number, baseMs: number, capMs: number): number {
  const ceiling = Math.min(capMs, baseMs * 2 ** attempt);
  return Math.random() * ceiling;
}

function sleep(ms: number, signal?: AbortSignal): Promise<void> {
  return new Promise((resolve, reject) => {
    if (signal?.aborted) return reject(signal.reason);
    const timer = setTimeout(() => {
      signal?.removeEventListener('abort', onAbort);
      resolve();
    }, ms);
    const onAbort = () => {
      clearTimeout(timer);
      reject(signal!.reason);
    };
    signal?.addEventListener('abort', onAbort, { once: true });
  });
}

async function withRetry<T>(fn: () => Promise<T>, opts: RetryOptions): Promise<T> {
  for (let attempt = 0; ; attempt++) {
    try {
      return await fn();
    } catch (err) {
      const isLast = attempt + 1 >= opts.maxAttempts;
      if (!(err instanceof RetryableError) || isLast) throw err;

      const backoff = fullJitter(attempt, opts.baseMs, opts.capMs);
      // Respect the server's hint if it asked for more time than we planned.
      const delay = Math.max(backoff, err.retryAfterMs ?? 0);
      await sleep(delay, opts.signal);
    }
  }
}
```

Then classify failures at the call site. That's the only place that knows what an error means:

```typescript
function parseRetryAfter(header: string | null): number | undefined {
  if (!header) return undefined;
  const seconds = Number(header);
  if (Number.isFinite(seconds)) return seconds * 1000;
  const date = Date.parse(header);
  return Number.isNaN(date) ? undefined : Math.max(0, date - Date.now());
}

const RETRYABLE_STATUS = new Set([408, 429, 502, 503, 504]);

async function getJson(url: string, signal: AbortSignal): Promise<unknown> {
  let res: Response;
  try {
    res = await fetch(url, { signal });
  } catch (err) {
    if (signal.aborted) throw err;          // our deadline, not a transient fault
    throw new RetryableError(`network error: ${(err as Error).message}`);
  }
  if (RETRYABLE_STATUS.has(res.status)) {
    throw new RetryableError(`HTTP ${res.status}`, parseRetryAfter(res.headers.get('retry-after')));
  }
  if (!res.ok) throw new Error(`HTTP ${res.status}`); // 400, 401, 404...: retrying won't help
  return res.json();
}

const signal = AbortSignal.timeout(10_000);
const profile = await withRetry(
  () => getJson('https://api.example.com/profile/42', signal),
  { maxAttempts: 4, baseMs: 100, capMs: 5_000, signal },
);
```

A few decisions in this code matter more than the backoff formula:

- **Allow-list retryable errors.** Timeouts, connection resets, 429, 502, 503 and 504 are usually transient. A 400 or 404 will fail the same way every time, so retrying it just wastes capacity. Retrying a 500 is a judgement call: it is often a deterministic bug, not a transient fault.
- **One deadline for the whole operation.** The same `AbortSignal` bounds every attempt and every sleep. Without it, four attempts with their own timeouts plus backoff can keep a user waiting far longer than any single timeout suggests.
- **Honour `Retry-After`.** When a server or rate limiter tells you how long to wait, that's better information than your formula. The header can be a number of seconds or an HTTP date, and the parser handles both.
- **Only retry idempotent operations.** Retrying a `GET` is safe. Retrying a `POST /payments` after a timeout can charge someone twice, unless the API supports idempotency keys and you send the same key on every attempt.

Before hand-rolling this, check your client library. The AWS SDKs, Google Cloud client libraries, gRPC and most mature HTTP clients already ship with jittered backoff. Write your own only when you're calling something that has none.

## Where retry logic should live

Because retries multiply across layers, choose one layer to own them for each call path and turn them off elsewhere.

### Clients (mobile, web, SDKs)

Clients sit at the edge, often in very large numbers, and you can't hot-fix them once they ship. That makes them the most dangerous place to get retries wrong and the most important place to use full jitter, sensible caps and `Retry-After`. Ship aggressive client retries, and every app install that reconnects after an incident becomes part of the herd.

### Service mesh and proxies

Between services, a proxy can own retries consistently. Envoy's router uses a fully jittered exponential backoff by default. Its [router documentation](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http%5Ffilters/router%5Ffilter?ref=stalin-pereira.com) lists a 25ms base and a cap of ten times the base. You can tune it per route:

```yaml
route:
  cluster: orders
  timeout: 3s
  retry_policy:
    retry_on: "connect-failure,reset,retriable-status-codes"
    retriable_status_codes: [503]
    num_retries: 2
    per_try_timeout: 1s
    retry_back_off:
      base_interval: 0.1s
      max_interval: 1s
```

If the mesh retries, application code on that path shouldn't retry as well. Pick one owner.

### gRPC

gRPC supports retries in the service config, with exponential backoff and an optional client-side retry throttle (see the [gRPC retry guide](https://grpc.io/docs/guides/retry/?ref=stalin-pereira.com)):

```json
{
  "methodConfig": [{
    "name": [{ "service": "orders.v1.OrderService" }],
    "retryPolicy": {
      "maxAttempts": 4,
      "initialBackoff": "0.1s",
      "maxBackoff": "1s",
      "backoffMultiplier": 2,
      "retryableStatusCodes": ["UNAVAILABLE"]
    }
  }],
  "retryThrottling": { "maxTokens": 10, "tokenRatio": 0.1 }
}
```

The `retryThrottling` block is a retry budget. Failures drain tokens and successes refill them. When the token count falls below half of `maxTokens`, the client stops retrying until the server recovers. That's the piece most hand-written retry loops are missing.

### Queues and background jobs

Queues don't give you exponential backoff for free. An SQS visibility timeout, for example, is a fixed delay before a message becomes visible again. For a growing delay, the consumer has to extend it explicitly on each failed receive (for SQS, with `ChangeMessageVisibility` based on the receive count), and after a few tries the message should go to a dead-letter queue. Background jobs can afford much larger bases and caps than interactive requests.

## Backoff is necessary, not sufficient

Backoff controls how fast a single client comes back. It doesn't limit the total load from many clients, and it doesn't stop a cache stampede on its own. A few companions close those gaps:

- **Retry budgets.** Cap retries as a fraction of traffic, or as a fixed rate per process, so that during a broad outage retries fade out instead of adding load. The gRPC throttle above and the AWS SDKs' retry quota both work this way.
- **Circuit breakers.** When a dependency is clearly down, stop calling it for a while and fail fast. Backoff decides when to try again. A circuit breaker decides whether to try at all. The two work best together; see [implementing circuit breakers with Resilience4j and Polly](https://stalin-pereira.com/implementing-circuit-breakers-with-resilience4j-polly/).
- **Request coalescing for caches.** When a hot cache key expires, the herd comes from first attempts, not retries. Let one request rebuild the value while the others wait for it ("single-flight"), serve the stale value during the refresh, and add a little randomness to TTLs so related keys don't all expire together.
- **Server-side load shedding.** Servers should reject excess work early and cheaply with a 429 or 503 and a `Retry-After` header. Well-behaved clients then do the right thing automatically.

The Amazon Builders' Library article [Timeouts, retries, and backoff with jitter](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/?ref=stalin-pereira.com) covers these interactions in more depth and is worth reading in full.

## When not to retry at all

- **Non-idempotent writes without idempotency keys.** A duplicate side effect is worse than a visible error.
- **Deterministic failures.** Validation errors, auth failures, and "not found" won't change on the next attempt.
- **Deep in a call chain where something above already retries.** Let the outermost layer own the retry and fail fast underneath.
- **When the latency budget is already spent.** If the user-facing deadline has 50ms left, a retry with backoff can't succeed in time. Return the error, or a fallback.

## Checklist

- Use exponential backoff with **full jitter** (or decorrelated jitter), not fixed intervals and not additive jitter.
- Set a **cap** on the delay and a **maximum number of attempts**. For interactive traffic, two or three retries is usually plenty.
- Bound the whole operation with **one deadline** that covers every attempt and every sleep.
- Retry only an **allow-list** of transient errors, and only **idempotent** operations (or ones protected by idempotency keys).
- Honour **`Retry-After`** when the server sends it.
- Have **one layer** own retries on each call path, and add a **retry budget** or circuit breaker so retries fade out during broad outages.
- Emit a metric for every retry. A rising retry rate is often the first sign that a dependency is in trouble.