Circuit Breakers, Bulkheads & Fallbacks

Three patterns of failure isolation

Level 1's resilience-primitives lesson sketched the circuit-breaker state machine (Closed to Open to Half-Open) as a first pass. This lesson credits that and goes past the single breaker to what senior answers actually hinge on: how the breaker and the bulkhead cover different halves of the same failure, and how a service mesh configures both. Timeouts and retries stop one slow call from hanging forever; when a dependency is broadly failing you instead want to stop calling it at all, contain the damage to one part of your service, and serve something useful instead of an error. That is circuit breakers, bulkheads, and fallbacks: the three patterns of failure isolation.

Circuit breaker

A circuit breaker is a state machine wrapped around a dependency that trips like an electrical breaker so you stop sending requests into a failure.

                trip on failure rate
     CLOSED  ---------------------------->  OPEN
       ^                                      |
       |                                      | after cooldown
       | probe succeeds                       v
       +---------------- HALF-OPEN <----------+
                probe fails -> back to OPEN

• Closed is normal: requests flow, and the breaker counts failures over a rolling window.
• Open trips when the failure rate crosses a threshold over a rolling window. Level 1 used the canonical "half of the last 20 calls"; real configs harden that by also gating on a minimum request volume (so a 2-of-3 blip cannot trip a breaker) and by pairing the error-rate threshold with a slow-call-rate threshold (so climbing latency alone can trip it before errors even appear). In Open state calls fail immediately without touching the dependency, which protects your callers from waiting on timeouts and sheds all load off the sick dependency so it can recover instead of being pinned down.
• Half-Open starts after a cooldown (say 5 seconds). The breaker lets a small number of trial requests through. If they succeed, it closes; if they fail, it re-opens and waits again.

The key insight is that failing fast is a feature. A breaker in Open state gives an instant error, which is far better than a client waiting 500 ms for a timeout on every request, and it is the only thing that lets an overloaded dependency drain its queue.

Bulkhead

A bulkhead isolates resources per dependency, named after ship compartments that stop one flooded section from sinking the vessel. If your service calls Dependencies A, B, and C from a single shared thread pool of 200 threads, and C gets slow, requests to C hold threads until they time out. Enough slow C calls and all 200 threads are stuck in C, so calls to A and B, which are perfectly healthy, get no threads and fail too. One sick dependency starved the others. The fix is to give each dependency its own bounded pool (for example 60 threads for A, 60 for B, 40 for C). Now a C brownout can consume at most C's 40 threads; A and B keep serving. Bulkheads convert a total outage into a partial one.

Fallbacks

Fallbacks answer "what do we serve when the dependency is unavailable?" Options, in order of preference: return cached or slightly stale data; return a sensible default; or gracefully omit the feature. The rule is that only non-critical dependencies should be fallback-able. You cannot fall back on the payment authorization, but you absolutely can fall back on the "customers also bought" recommendations by rendering the page without them. The product still sells.

Interview nuance: breakers and bulkheads solve different halves of the same problem, and strong answers use both. The breaker decides whether to call a dependency based on its recent health; the bulkhead bounds how much of your resources any one dependency can ever consume, even before the breaker trips. Without the bulkhead, a dependency that is slow but not yet failing enough to trip the breaker can still exhaust your shared pool. Envoy and service meshes provide both as config (outlier detection for breaking, circuit-breaker connection/request limits for bulkheading).

Recap: circuit breakers move Closed to Open to Half-Open to fail fast and let a sick dependency recover; bulkheads give each dependency a bounded pool so one cannot starve the others; fallbacks serve stale, default, or omitted content, but only for non-critical dependencies.