LLM Inference Serving (GPU Economics)

The GPU is the budget, and the KV cache is the cap

Self-hosting an LLM means the GPU is the budget, and inference efficiency is the difference between serving 5 and 50 requests per GPU. The interview tests whether you understand why LLM serving is unlike serving a stateless web service. The answer is the KV cache and batching.

Why generation is memory-bound

A transformer generates one token at a time. For each new token it attends over all previous tokens, so it caches the key and value tensors of every prior token: the KV cache. That cache grows with sequence length and must stay in GPU memory (HBM) for the whole request. A single long-context request can hold gigabytes of KV cache. Since GPU memory is fixed (say 80GB on an H100, minus the model weights), the KV cache, not compute, is what caps how many requests you can run at once. Interview nuance: this is why "just batch more" is not free. Every concurrent request reserves KV memory.

PagedAttention and continuous batching

Classic serving pre-allocates a contiguous KV block per request sized to the max length, so a request that generates 50 tokens still reserves memory for thousands. That fragmentation wastes most of the KV memory. PagedAttention (the core vLLM idea) treats KV cache like virtual memory: it allocates in small fixed pages on demand and maps them with a page table. Waste drops to near zero, so you fit far more concurrent requests in the same GPU, which directly raises throughput.

Static batching waits to collect a batch, runs it to completion, then starts the next, so a batch runs only as fast as its slowest (longest) sequence and finished sequences idle the GPU. Continuous batching schedules at the token level: as soon as one sequence finishes it is evicted and a queued request joins the running batch mid-flight. The GPU stays saturated. Combined with paging, this is the single biggest throughput win in modern serving and is why vLLM, TGI, and TensorRT-LLM all do it.

The latency metrics you must name

Time to first token (TTFT)  = prefill: process the whole prompt once (compute-bound)
Inter-token latency (ITL)   = decode: one token at a time    (memory-bound)
Total latency = TTFT + ITL x output_tokens
Throughput    = total tokens/sec across all concurrent requests

TTFT is dominated by prompt length (prefill). ITL is the streaming speed the user feels. Throughput and latency trade off: larger batches raise throughput but each request's ITL rises because the GPU is shared. Chunked prefill (splitting a long prompt so it interleaves with ongoing decodes) and prefill/decode disaggregation (separate GPU pools for the compute-bound prefill and memory-bound decode) let you protect TTFT without starving decode.

The other levers

Quantization (INT8, FP8, AWQ) shrinks weights and KV cache so more fits and math is faster, at a small accuracy cost. Tensor and pipeline parallelism shard a model too big for one GPU across many. Prefix caching reuses the KV of a shared system prompt across requests so you prefill it once. Speculative decoding drafts several tokens with a small model and verifies them with the big one to cut ITL. Autoscaling keys on GPU utilization and queue depth, not CPU.

Recap: LLM serving is capped by KV-cache memory, so use PagedAttention to kill fragmentation and continuous batching to keep the GPU saturated; reason in TTFT (prefill) vs inter-token (decode) vs throughput; and add quantization, parallelism, prefix caching, and speculative decoding to stretch a fixed GPU fleet.