Timeouts, Retries, Backoff & Circuit Breakers

How one slow dependency becomes an outage

A distributed system fails one dependency at a time, and the way one failure becomes an outage is almost always the caller mishandling a slow or broken downstream. The client-side call policy is your primary defense, and it has four moving parts: timeouts, retries, circuit breakers, and isolation.

Timeouts

Every network call must have one. The default in most HTTP clients is infinite or 30+ seconds, which is a trap: when a downstream stalls, your threads (or async slots) block waiting, the pool drains, and you stop serving healthy requests too. This is the classic cascading failure. Set the timeout from the downstream's SLO (for example, if it promises p99 of 50ms, time out at maybe 150ms), and propagate a deadline down the call chain. If the top-level request has a 300ms budget and 200ms is already spent, the next hop should be told it has 100ms left, not handed a fresh 150ms. gRPC deadlines and context propagation do this for you; without it, downstreams do work for a client that already gave up.

Retries

A retry can turn a transient blip into a success, but only under two conditions. First, the operation must be idempotent or the error safely retryable (a timeout on a non-idempotent POST might have already charged the card). Use idempotency keys so a retried write dedupes. Second, retries must have exponential backoff with jitter. Without backoff, thousands of clients retry in lockstep the instant a service hiccups, creating a synchronized thundering herd that keeps the service down (a "retry storm"). Backoff spreads them out; jitter (randomizing the delay) breaks the synchronization. Cap the total with a retry budget: allow retries only up to, say, 10% of request volume, so a widespread failure cannot multiply your load 3x and turn a partial outage into a total one.

Interview nuance: "Retries make it more reliable" is only half true. The senior answer names the failure mode retries cause (retry amplification) and the three guards: idempotency, backoff-with-jitter, and a retry budget.

Circuit breaker

When a downstream is genuinely down, retrying at all is waste that adds load. A circuit breaker tracks the recent failure rate and has three states. Closed: calls flow normally. When failures cross a threshold (for example 50% of the last 20 calls), it trips to Open: calls fail fast immediately without touching the network, giving the downstream room to recover and freeing your threads. After a cool-down it goes Half-open: it lets a trickle of trial calls through, and if they succeed it closes, if they fail it re-opens. This converts a slow, thread-eating failure into a fast, cheap one.

Isolation and fallback

Bulkheads give each dependency its own bounded connection pool or thread pool, so one slow dependency drowns only its own bulkhead instead of every thread in the process (the pattern that named the Hystrix library). When a call fails fast, degrade gracefully: serve a cached value, a default, or a partial response rather than an error.

Closed --failures over threshold--> Open --cool-down--> Half-open --trial ok--> Closed
   ^                                                         |
   +---------------- trial fails ----------------------------+

Recap: Give every call a propagated deadline, retry only idempotent errors with backoff, jitter, and a budget, trip a circuit breaker to fail fast when a dependency is down, and isolate with bulkheads so one slow dependency cannot drain the whole caller.