Capacity Planning & Back-of-Envelope Sizing

From "how much traffic" to "how many machines"

Capacity planning is turning traffic numbers into machine counts you can defend on a whiteboard. The core engine is Little's Law: in a stable system, the average number of requests in flight equals arrival rate times average time-in-system.

  L (concurrency) = lambda (RPS) x W (latency in seconds)

If you serve 50,000 RPS and each request spends 100ms (0.1s) being processed, average concurrency is 50000 x 0.1 = 5000 requests in flight simultaneously. That is the real sizing number: not RPS, but concurrent work. If one instance can hold ~250 concurrent requests before its own latency degrades, you need 5000 / 250 = 20 instances just to hold steady-state concurrency.

Never size to 100%

You never size to 100% of steady state, for two reasons rooted in queueing theory. First, utilization and latency are not linear. As utilization approaches 100%, queue length and wait time explode toward infinity (the 1/(1-rho) term): going from 70% to 90% utilization can double or triple your p99. Second, you need slack to absorb bursts and GC/pause jitter. So you target 50 to 70% utilization: 20 instances at a 70% target becomes 20 / 0.7 = ~29 instances.

Redundancy math

You must survive failure of a whole availability zone, so you spread instances across (typically) 3 AZs and size so that losing one AZ still leaves enough capacity. This is N+1 thinking at the AZ level. If 29 instances serve peak at target utilization across 3 AZs, losing one removes a third of the fleet; to keep the surviving two AZs at or below target after a zone loss, provision ~50% more, so 2/3 of the fleet still covers 100% of peak. So ~29 becomes ~44 instances (roughly 15 per AZ).

Finally, peak-to-average ratio sets autoscaling bounds and the reserved-vs-on-demand mix. Buy reserved/savings-plan capacity for the always-on baseline (cheapest per hour), on-demand for the predictable daily peak, and spot for burst or batch (cheapest but pre-emptible). You do not reserve for peak, because peak is a small fraction of the day.

Interview nuance: the fastest way to sound junior is to divide RPS by "requests per second per server" and stop. The fastest way to sound senior is to (1) convert to concurrency with Little's Law, (2) apply a utilization target and say why (queues explode near 100%), and (3) add explicit N+1 AZ redundancy. State the estimation chain out loud: DAU x actions/user/day / 86,400s x peak multiplier = peak RPS. Interviewers grade the method, not the exact number.

Recap: size with Little's Law (concurrency = RPS x latency), divide by a 50 to 70% utilization target because queues blow up near 100%, add N+1 AZ redundancy so losing a zone stays above peak, and split the resulting capacity across reserved/on-demand/spot by peak-to-average ratio.