Backpressure, Flow Control & Load Shedding

What happens when traffic exceeds capacity

Every system has a maximum sustainable throughput. The question is what happens when arriving traffic exceeds it. The wrong answer is "queue it all and hope," because that quietly trades a latency problem for a crash. The right answer is a deliberate overload strategy built from backpressure, bounded queues, and load shedding.

Backpressure and the unbounded-queue trap

Backpressure is the signal that flows upstream telling producers to slow down. In a well-designed pipeline, a full downstream buffer stops the upstream from producing, all the way back to the source. TCP flow control does this at the socket level; reactive stream libraries (Reactive Streams, gRPC flow control) do it at the application level; a bounded queue does it implicitly, because a producer that cannot enqueue must block or drop. The enemy of backpressure is the unbounded queue. It looks like it is absorbing the spike, but it is really accumulating latency (a request that waits 30 seconds in a queue is useless, the user left) and memory, until the process runs out of heap and OOM-crashes, taking down everything including the in-flight work that was fine. Bound every queue.

Why you cannot run hot

As utilization (rho) approaches 100%, queue length and wait time do not rise linearly, they explode. A rough mental model from M/M/1 queues: average time in system scales like 1 / (1 - rho). At 50% utilization latency is roughly 2x the service time; at 90% it is 10x; at 99% it is 100x. This is why you provision to run at 60 to 70% and treat the last 30% as headroom for spikes, not capacity to sell. A system run at 95% "efficient" utilization has a brutal tail.

Interview nuance: If asked "why not just run at 100% utilization, isn't that efficient," answer with the 1/(1-rho) intuition. Utilization is bought with latency, and near saturation the price is unbounded.

Load shedding: reject early, on purpose

Load shedding (admission control) is the deliberate choice to reject some work so the rest survives. When you are over capacity, it is far better to reject early with a 429 or 503 at the edge than to accept a request, let it sit in a queue, and time it out after doing partial work. Rejecting early is cheap and preserves latency for accepted requests; queue-and-timeout burns capacity on work nobody will use (this is "goodput" collapsing even as "throughput" stays busy). Shed at the front door, before you have invested resources.

Do it with real tools: concurrency limits (cap in-flight requests, the most robust knob because it directly bounds Little's Law's L), token-bucket rate limiters (smooth bursts to a sustainable rate), and adaptive concurrency (algorithms like Netflix's that watch latency and dynamically lower the limit as latency rises, no hand-tuned magic number). Pair shedding with prioritization: shed low-value traffic first (batch, retries, free tier) so critical traffic (checkout, paying users, health of the system) survives. And drop stale work: if a request has already exceeded its deadline while queued, discard it instead of processing it, because the caller has already given up.

arrivals --> [admission control] --accept--> [bounded queue] --> workers
                    |                              |
                  reject                       drop if stale/
                429/503                        past deadline

Recap: Bound every queue and let backpressure propagate, run below saturation because latency explodes as utilization nears 100%, and when overloaded reject early with 429/503, prioritize critical traffic, and drop stale requests instead of letting an unbounded queue hide the overload until an OOM.