Load Shedding, Adaptive Concurrency & Backpressure

Systems die by accepting more than they can finish

Level 1's "Backpressure, Flow Control & Load Shedding" lesson established the primitives: bound every queue, let backpressure propagate, and reject early because latency explodes as utilization nears 100%. This lesson takes those as given and leads with the failure they exist to prevent, the retry-storm death spiral, plus the two senior moves for beating it, adaptive concurrency limits and brownout. Rate limiting caps one client's demand; overload protection is what you reach for when total legitimate demand simply exceeds your capacity, or a dependency slows and requests pile up. The goal is blunt: at 150% of capacity, stay up and keep serving the most important traffic, instead of trying to serve everything and serving nothing.

Understand the failure mode first. A server has finite concurrency. When arrival rate exceeds completion rate, in-flight requests and queues grow, each request waits longer, latency climbs, clients hit timeouts and retry (often amplifying load 3x), memory for queued work grows, and eventually the box GCs itself to death or OOMs. This is the congestion collapse / retry-storm death spiral: throughput actually drops toward zero under increasing load. The defining mistake that enables it is the unbounded queue, which hides overload by accepting work it will never complete in time until memory runs out.

Load shedding: reject early, by priority

A request you reject in 1ms costs almost nothing; a request you accept, queue for 5s, then fail costs capacity you needed for good traffic. So you shed before collapse, at a threshold below 100%, and you shed the right traffic. Priority-aware shedding classifies traffic (health checks and paying-customer writes are critical; bulk exports, retries, and best-effort reads are droppable) and drops low-priority first, so the checkout path survives while a recommendation call is dropped.

Adaptive concurrency limits

A static "max 500 concurrent" is wrong the moment your dependency's latency changes: at 50ms per request 500 concurrency is fine, at 500ms it is 10x too much. Instead, discover the limit dynamically the way TCP congestion control does. By Little's Law, concurrency = throughput * latency; a system probes by raising its concurrency limit while latency stays flat and backing off when latency rises (a gradient / TCP-Vegas style loop, as in Netflix's adaptive concurrency limiter). The limit tracks the real, current capacity with no operator-tuned magic number.

Backpressure

Refuse upstream when you are full, so pressure propagates back to the source instead of accumulating in you. Use bounded queues that reject (or return a fast 503) when full. Propagate deadlines: pass a per-request deadline through the call chain and drop any request whose deadline has already passed, since finishing already-dead work is pure waste.

Interview nuance: graceful degradation / brownout is the senior move. Under overload you can shed features: serve a cached or partial response, skip the personalization call, drop the recommendation carousel, return the core page. Combine with retry hygiene (exponential backoff plus jitter, and circuit breakers) and you break the retry storm at both ends.

demand ---> [ admission: shed low-priority first if over threshold ]
             |
             v
        [ bounded queue: reject/503 when full, drop past-deadline ]
             |
             v
        [ worker pool: adaptive concurrency = probe via Little's Law ]
   overload -> brownout: cached/partial responses, drop optional features

Recap: shed early and by priority, replace static thresholds with adaptive concurrency limits derived from Little's Law, bound every queue and drop past-deadline work so latency cannot explode, and brown out optional features rather than failing everything: never an unbounded queue.