Horizontal vs Vertical Scaling & Stateless Services

Scale up hits a wall; scale out needs statelessness

There are two ways to serve more traffic. Scale up (vertical) means a bigger box: more cores, more RAM, faster disks on the same machine. Scale out (horizontal) means more boxes behind a load balancer. Scale-up is the easy first move because it needs no code changes, but it hits a wall fast: hardware has a top SKU, price scales super-linearly past commodity sizes (a 128-core box costs far more than 2x a 64-core box), and one box is a single failure domain. When it dies, you are fully down. Scale-out is the web-tier default precisely because it dodges all three: commodity nodes are cheap, you add capacity linearly, and losing one node loses only 1/N of capacity.

The catch, and the whole point of this lesson: you cannot load-balance servers that hold local state. If a node keeps the user's session in its own process memory, then request 1 lands on node A (which now holds the session), and request 2 might land on node B, which has never heard of that user. The user appears logged out. Worse, when node A dies, every session it held is gone. The load balancer can only freely spread requests if any node can serve any request, which means nodes must be stateless.

Externalizing state

• Sessions: move them to Redis or Memcached, or make them stateless entirely with a signed JWT the client carries. Now any node validates the token or reads the session store, and node death loses nothing.
• Uploaded files / user assets: to object storage (S3, GCS), never local disk.
• Durable data: to the database, which is a separate scaling problem.

Once state is externalized, nodes become cattle, not pets. A pet is a hand-tuned server with a name you nurse back to health. Cattle are interchangeable and disposable: provisioned from an immutable image or IaC (a baked AMI, a container, Terraform), and when one misbehaves you kill it and boot a replacement rather than debugging it live. Autoscaling groups, Kubernetes deployments, and rolling deploys all assume this.

Interview nuance: do not over-apply "scale out everything." Scale-up still wins for tiers that are genuinely hard to shard: a single-writer relational database, an in-memory analytics engine, anything where the working set must be co-located. There you buy the big box and defer sharding until write throughput or dataset size truly forces it. The honest framing: scale-out for the stateless web/app tier, scale-up (then shard) for the stateful data tier.

  scale UP (vertical)            scale OUT (horizontal)
  +-------------+                +----+  +----+  +----+
  |  bigger box |     vs         | n1 |  | n2 |  | n3 |  ... n500
  +-------------+                +----+  +----+  +----+
  1 failure domain,                 \      |      /
  hard ceiling                    [ shared state: Redis / DB / S3 ]

Recap: scale-out is the web-tier default because it beats the cost, ceiling, and single-failure-domain limits of scale-up, but it only works once nodes are stateless (session and file state externalized to Redis/JWT/S3), turning servers into interchangeable cattle; scale-up still wins for hard-to-shard stateful tiers until you are forced to shard.