Service Discovery & Client vs Server-Side Load Balancing

Callers must learn addresses that change every minute

In a fleet that autoscales and redeploys constantly, a service's instances come and go every minute: IPs and ports change, instances are added under load and terminated on scale-in. So a caller cannot hardcode addresses, and it cannot rely on long-TTL DNS, because the moment an instance is terminated a stale address keeps receiving traffic and callers get connection errors. Service discovery lets a caller learn the current set of healthy addresses, and the second question is who makes the balancing decision: a central load balancer, or each client.

The service registry

• Self-registration: each instance registers itself on startup and sends periodic heartbeats (Consul, etcd, Netflix Eureka). If heartbeats stop, the registry marks it gone. On graceful shutdown it deregisters.
• Platform-managed: the orchestrator maintains it for you. In Kubernetes, a Service is a stable name/VIP, and the control plane keeps its Endpoints / EndpointSlices in sync with the pods that pass their readiness probe.

Health-based removal keeps discovery honest. The registry advertises only instances that pass active health checks or are heartbeating, combined with readiness so a new instance receives traffic only once warm. The number that matters is propagation speed: how fast a terminated or failing instance actually leaves every caller's view. With short check intervals plus fast registry watch/push, a bad instance is out of rotation within seconds; with long DNS TTLs it can be minutes, which is the failure mode to avoid.

Where the balancing decision happens

• Server-side load balancing: clients hit one stable VIP or DNS name and a dedicated load balancer (ALB/NLB, Envoy, Nginx) picks a backend. Clients stay dumb and simple, and control is central. The cost is an extra network hop and a component you must scale and keep HA.
• Client-side load balancing: the client fetches the healthy instance list from the registry (or a mesh sidecar) and picks a backend itself (gRPC client-side LB, a sidecar Envoy). This removes the extra hop and enables smart, locality-aware policies (prefer same-zone, least-request with local load view). The cost is complexity pushed into every client and a hard dependency on fast registry propagation.

A service mesh (Istio or Linkerd, Envoy sidecars) is the popular middle ground: client-side benefits (no central-LB hop, locality, per-request balancing, retries, mTLS) with central configuration. The price is real operational complexity (a control plane and a sidecar per pod).

Interview nuance: the discriminator is where you want complexity to live. Central LB = simple clients, extra hop, one scaling choke point. Client-side/mesh = no hop and smart routing, but complexity and propagation risk in every caller. A strong concrete answer: Kubernetes with a mesh for a polyglot fleet, or gRPC client-side LB backed by etcd for a gRPC-heavy one, with short health-check intervals so bad instances leave rotation within seconds.

  registry (Consul/etcd/Eureka  |  k8s Endpoints via readiness)
       ^ register/heartbeat            ^ controller keeps in sync
  server-side:  client -> [ VIP/LB ] -> backend        (1 extra hop, central control)
  client-side:  client (has list) ---> backend         (no hop, smart local policy)

Recap: callers learn healthy addresses from a service registry (self-registration with heartbeats, or Kubernetes Endpoints tied to readiness), unhealthy instances leave rotation in seconds; server-side LB keeps clients simple at the cost of a hop and a central component, client-side/mesh removes the hop and adds locality at the cost of per-client complexity, and the classic wrong turn is hardcoded IPs or long-TTL DNS that keeps sending traffic to terminated instances.