Load Shedding & Graceful Degradation

Load shedding protects you from too many clients

Levels 1 and 4 already covered the shedding mechanics (reject early at the edge with 429/503, bound every queue, prioritize by request class, discover the limit with adaptive concurrency); this lesson recaps those in one pass and leads with the two ideas that decide whether shedding actually saves you: goodput versus throughput, and metastable failure. Circuit breakers protect you from a sick dependency; load shedding protects you from too many clients. When demand exceeds capacity you have two choices: try to serve everyone and serve no one (collapse), or deliberately reject some requests so the rest succeed. Controlled partial service beats total collapse, every time.

Goodput, not throughput

Throughput is requests you process; goodput is requests you process successfully and in time. Under overload these diverge sharply. Imagine a service that maxes out at 10k QPS of goodput. Push 20k QPS at it with no shedding and throughput climbs while goodput falls, because the machine spends its CPU on context switches, GC, and requests that will time out before the client sees them. You are doing 20k QPS of work and delivering maybe 3k useful responses. The extra 17k is pure waste that actively harms the 3k. Maximizing goodput means throwing away the doomed work early so the machine's capacity goes to requests that can actually complete.

The mechanics, recapped in one pass

Levels 1 and 4 own the how, so this is only the recap. Shed at the edge with 429 Too Many Requests or 503 Service Unavailable plus a Retry-After header before you spend work on a request; prioritize by request class so prefetch and batch jobs die before checkout and paying-customer writes; and cap in-flight work with bounded queues and adaptive concurrency limits (a TCP-Vegas-style controller such as Netflix's concurrency-limits) rather than a bigger queue, which adds only latency and eventually an OOM. With those assumed, the rest of this lesson is why they work: goodput, and the metastable trap they exist to break.

Metastable failures

  normal ---(trigger: traffic spike)---> overloaded
     ^                                       |
     |                                       | retries + full queues
     +------ (does NOT self-recover) --------+
              the trigger is GONE but the system stays down

A metastable failure is one that sustains itself after the original trigger is gone. A traffic spike pushes the system into overload; the overload causes timeouts; the timeouts cause client retries; the retries add more load than the original spike; and now even after the spike passes, the retry-driven load keeps the system saturated. Adding capacity often does not break the loop, because the retries scale up to consume it. The way out is to attack the feedback loop directly: shed load aggressively to drop goodput demand below capacity, and combine it with backoff and jitter on the clients so the retry wave dissipates. Sometimes you must shed almost everything briefly to let the queues drain, then ramp back.

Interview nuance: the wrong turn is "we will autoscale." Autoscaling is minutes-slow and cannot outrun a retry storm that doubles load in seconds, and if the bottleneck is a shared database, more app servers make it worse. Load shedding acts in milliseconds at the edge and is the only thing that reliably breaks a metastable collapse.

Recap: maximize goodput not throughput by discarding doomed work early; shed cheaply at the edge with 429/503 and Retry-After; prioritize by request class; use admission control and bounded concurrency instead of unbounded queues; and break metastable failures with aggressive shedding plus client backoff, because autoscaling is too slow.