Consumer Groups, Rebalancing & Scaling

How Kafka scales reads

A consumer group is how Kafka scales reads. All consumers sharing a group.id cooperatively divide a topic's partitions, and Kafka guarantees each partition is assigned to at most one consumer in the group. A group with 4 consumers over a 12-partition topic gives each consumer 3 partitions. Two different groups ("ranking" and "analytics") each get the full stream independently: that is how one topic fans out to many pipelines.

The immediate ceiling: group parallelism is capped by partition count. With 12 partitions, a 13th consumer sits idle. This is the number one scaling mistake: adding consumers past the partition count does nothing. You scale reads by having enough partitions in the first place.

Offsets and delivery semantics

Each consumer tracks its position per partition as a committed offset in the internal __consumer_offsets topic. When and how you commit decides your delivery guarantee:

• Auto-commit (every 5s) commits on a timer regardless of whether processing finished, so a crash after commit but before processing loses messages, and a crash before commit reprocesses. Convenient and wrong for anything that matters.
• Manual commit after processing (commit only once the side effect is durably done) gives at-least-once: crash after processing but before committing and you reprocess a duplicate. The sane default.
• Committing before processing gives at-most-once and silently drops work on a crash.

Because the safe choice is at-least-once, your consumer handlers must be idempotent. Duplicates are guaranteed around every crash and every rebalance.

Rebalancing, the sharp edge

When a consumer joins, leaves, or is presumed dead (misses heartbeats), the group rebalances: partitions are reassigned. The classic "eager" protocol is stop-the-world: every consumer revokes all its partitions, then the group reassigns from scratch, so the entire group stops for the rebalance (hundreds of ms to seconds). A deploy that restarts 30 consumers one by one can trigger 30 rebalances, each a latency and duplicate spike.

Group "ranking", topic 6 partitions, 3 consumers:
  C1 -> p0,p1   C2 -> p2,p3   C3 -> p4,p5
C3 dies -> rebalance -> C1 -> p0,p1,p4   C2 -> p2,p3,p5
 Eager: ALL stop, revoke everything, reassign. Cooperative: only p4,p5 move.

Cooperative (incremental) rebalancing revokes only the partitions that must move, so consumers keep processing unaffected partitions. Static group membership (group.instance.id) lets a restarting consumer rejoin with its old assignment within session.timeout.ms, so a rolling deploy causes no rebalance at all. And KIP-848 (the new consumer-group protocol, GA in Kafka 4.0) moves assignment computation to the broker-side coordinator and makes rebalances fully incremental, removing the stop-the-world join barrier. Tuning session.timeout.ms/heartbeat.interval.ms sensibly (e.g., 45s/3s) avoids spurious rebalances from a GC pause.

Interview nuance: the health/scaling signal is consumer lag (latest offset minus committed offset). Rising lag means you are falling behind; autoscale consumers on lag, but only up to the partition count, and alert on it. Do not scale on CPU alone.

Recap: a consumer group splits partitions one-per-consumer so group size is capped by partition count; offset-commit timing sets the delivery guarantee, and commit-after-process gives at-least-once so handlers must be idempotent; rebalancing is stop-the-world in the eager protocol but made incremental by cooperative rebalancing, static membership, and KIP-848; and consumer lag is the metric you scale and alert on.