Retries, Dead-Letter Queues & Backpressure

Failure handling decides whether your pipeline degrades or wedges

A consumer that calls anything flaky (a third-party API, a downstream service) will hit failures. How you handle those failures decides whether your pipeline degrades gracefully or wedges completely.

Retries with backoff and jitter. Transient failures (a 503, a timeout, a throttle) should be retried, but naively retrying immediately in a tight loop turns a downstream blip into a self-inflicted DDoS. Use exponential backoff (wait 1s, 2s, 4s, 8s) plus jitter (randomize the delay) so a fleet of consumers that all failed at once do not retry in a synchronized thundering herd. Cap the attempts (say 5) so a permanently broken message does not retry forever.

Transient vs permanent errors. Classify the failure. A timeout or 429 is transient: retry it. A 400 "malformed payload" or a schema violation is permanent: retrying will never succeed, so send it straight to the dead-letter queue instead of burning 5 attempts. Blindly retrying permanent errors wastes capacity and delays the DLQ signal.

The dead-letter queue

When a message exhausts its retries (or fails permanently), you must not drop it silently and you must not let it block the stream. You route it to a dead-letter queue or topic: a separate destination holding failed messages with their error context and attempt count. The DLQ needs three things to be useful: alerting (DLQ depth greater than zero pages someone), inspection (you can read why each message failed), and redrive (tooling to replay fixed messages back onto the main topic after you deploy a fix). A DLQ with no alerting is just a place data goes to die.

Head-of-line blocking, the core Kafka trap

A Kafka partition is a strictly ordered log, and a consumer processes it in order, one offset at a time. If message at offset 100 keeps failing and you retry it in place, you cannot advance to offset 101 without either committing past the failure (losing it) or blocking forever. One poison message stalls the entire partition and everything behind it. This is head-of-line blocking, and it is the number one wrong turn in async design.

  partition:  ... 98  99  [100 FAILS] 101  102  103 ...
                            ^ retrying in place
                            everything behind 100 is stuck

The fix is to not retry in place on the ordered partition. Two standard patterns: (a) move the failed message to a retry topic (often a tiered set: retry-5s, retry-1m, retry-10m) with a delay, commit the original offset, and let the main partition flow; a separate consumer drains the retry topic after the delay. (b) After N retries, move it to the DLQ. Either way the main partition never blocks on one bad message. The tradeoff: moving a message off the ordered partition breaks strict ordering for that key, so this is for workloads where per-message success matters more than strict order, or where you accept reordering on failure.

Backpressure

When a consumer is slower than the producer, something has to give. With a pull-based log like Kafka, the consumer fetches at its own pace and simply falls behind; the durable log is the buffer, absorbing the backlog on disk (days of retention) instead of overflowing memory or dropping data. Consumer lag (how many messages behind the head you are) is the health signal, and you respond by autoscaling consumers on lag (up to the partition count, which caps parallelism). Contrast a push-based system with no buffer, where a slow consumer forces the producer to block or drop. Bounding in-flight work per consumer keeps memory stable while the log holds the overflow.

Interview nuance: if asked "what happens when your consumer can't keep up," the strong answer is "the durable log absorbs it as lag, I alert and autoscale on lag up to partition count, and I make sure a poison message goes to a retry topic or DLQ rather than blocking the partition." That covers both the slow-consumer and the bad-message failure modes in one breath.

Recap: retry transient errors with capped exponential backoff plus jitter, send permanent failures and exhausted retries to an alerted, redrivable DLQ, never retry in place on an ordered partition (use retry topics to avoid head-of-line blocking), and lean on the durable log as your backpressure buffer while autoscaling on consumer lag.