Latency, Throughput, Percentiles & Little's Law

Three numbers, one law, and a lying average

Three numbers describe how a system behaves under load, and confusing them is the fastest way to sound junior. Latency is how long one request takes. Throughput is how many requests complete per second (QPS). Concurrency is how many requests are in flight at once. They are not independent, and the glue between them is Little's Law.

Averages lie

Imagine 100 requests, 99 at 10ms and one at 2000ms. The mean is 30ms, which sounds healthy, but 1% of your users just waited two seconds. The number that matters is a percentile: p50 (median) is the typical request, p99 is the request that only 1 in 100 users beats, and p99.9 is the tail your biggest, most active users hit constantly. Report p50, p95, p99, and p99.9, and treat p99 as the user number, because a heavy user who makes 100 requests per page load will almost certainly hit your p99 on every single page.

Percentiles are not evenly spaced. The gap from p50 to p90 is usually small, and then the curve turns almost vertical:

Table

The tail is not a slightly worse average, it is a different order of magnitude. The mean row is the point of the lesson: it sits between p50 and p99 and describes neither.

Reading	Latency	Multiple of median	Who feels it
p50 (median)	10 ms	1x	The typical request. This is what makes a dashboard look healthy.
p90	15 ms	1.5x	Still flat. Nine in ten users cannot tell this from the median.
p99	200 ms	20x	The cliff. A user making 100 requests per page hits this on nearly every page load.
p99.9	2 s	200x	Rare per request, constant for your largest and most active accounts.
mean	30 ms	3x	Nobody. No real request took 30 ms.

That shape is why an SLO is written against a percentile and a threshold together ("p99 under 300 ms") rather than against an average. An average can improve while the tail gets worse.

Fan-out makes the tail common

Tail latency gets worse, not better, as you scale, because of fan-out. If one API request fans out to 20 backend calls and you must wait for all of them, your response is as slow as the slowest of the 20. Even if each backend has a clean 1% chance of a slow (p99) response, the probability that at least one of 20 is slow is 1 minus 0.99^20, roughly 18%. So a backend p99 becomes a frontend p82. Fan-out turns rare tails into common ones, which is why Google's "tail at scale" work pushes techniques like hedged requests (send a duplicate after the p95 mark, take the first to answer).

Little's Law

L = arrival_rate x latency, where L is the average number of requests concurrently in the system. If you serve 2000 QPS and each request takes 50ms (0.05s), then on average 2000 x 0.05 = 100 requests are in flight, so you need at least 100 units of concurrency (threads, connections, or async slots). Turn it around: if you have a fixed pool of 200 workers and latency creeps to 200ms, your ceiling is 200 / 0.2 = 1000 QPS, no matter how much traffic arrives. Little's Law is how you size pools and how you spot that rising latency is silently capping throughput.

Interview nuance: When asked "how many threads/connections do you need," reach for Little's Law out loud. concurrency = QPS x latency is a one-line answer that signals you can size a system rather than guess.

The measurement trap: coordinated omission

Many load testers send the next request only after the previous one returns. When the server stalls, the tester stalls with it and simply fails to send the requests that would have piled up, so those never get timed. The result badly understates the tail. Fix it by measuring against intended send time (record when a request should have started), or use tools like wrk2 or HdrHistogram that correct for it. Always aggregate with histograms, not by averaging per-node p99s, because you cannot average percentiles.

Recap: Averages hide the tail, p99 is the number users feel and fan-out makes it common, and Little's Law (L = arrival_rate x latency) sizes your concurrency and exposes when latency is capping throughput.