The Log: Topics, Partitions, and Offsets

Level 9: Streaming & Change Data Capturemedium26 minKafka log modelpartitions as the unit of parallelismoffsetsretentionhot-key skewGROUP BYMAX and AVG

The append-only log as a data structure: topics split into partitions, immutable offsets inside a partition, retention that eats the head, and the key choice that decides whether one partition carries the whole load.

An event stream is an append-only log

Forget queues for a moment. A Kafka topic is not a mailbox that empties when someone reads it. It is a log: writers append to the end, readers read forward from wherever they choose, and reading changes nothing. Ten different teams can read the same record without taking it away from each other, and any of them can go back and read it again tomorrow. That single property is why a log sits under so many data platforms.

A topic is split into partitions, and a partition is the actual log. Partitions are the unit of parallelism: a topic with 12 partitions can be read by up to 12 consumers in one consumer group working in parallel, and a topic with 1 partition can be read by only one consumer per group, no matter how much hardware you throw at it. That ceiling is per group, not per cluster. Any number of independent groups can read the same topic at the same time, which is the next lesson. When a producer writes a record it picks a partition, usually by hashing the record's key, so every record with the same key lands on the same partition.

Inside a partition, every record gets an offset: a monotonically increasing integer that is immutable and unique within that partition only. Offset 500 in partition 0 and offset 500 in partition 3 are unrelated records. Ordering is guaranteed inside a partition and nowhere else, which is the second half of why key choice matters. If you need all of one customer's events in order, key by customer id and they all queue up on one partition in order.

Table

One topic, three partitions, three independent logs. Offsets are per partition, so offset 101 names a different record in each row. Order is guaranteed down a row, never across rows.
partition	offset 100	offset 101	offset 102	offset 103
p0 (keys hashing to 0)	order A1	order A2	order A3	(next append)
p1 (keys hashing to 1)	order B1	order B2	(next append)	''
p2 (keys hashing to 2)	order C1	(next append)	''	''

One topic, three partitions, three independent logs. Offsets are per partition, so offset 101 names a different record in each row. Order is guaranteed down a row, never across rows.

Retention eats the head, so depth is a subtraction

A partition is not infinite. Retention deletes records from the head of the log once they are older than the configured window (or once the partition exceeds a size cap). That gives every partition two markers the broker reports:

log_start_offset: the oldest offset still on disk. It only ever moves forward, as retention deletes.
log_end_offset: the offset the next appended record will receive. It is one past the newest record.

So the number of messages a partition currently holds is log_end_offset - log_start_offset, never log_end_offset on its own. On a topic that has been running for a year, log_end_offset is a lifetime counter and the difference is what you could actually replay. Retention is also your replay window: a topic with retention_hours = 24 cannot be replayed from last week, no matter how badly you need it.

Hot partitions are a key-choice bug

If a key is not evenly distributed, its partition gets hammered while its siblings idle. That is a hot partition, and it is the most common junior-reachable Kafka failure. The consumer that owns the hot partition falls behind on its own while everyone else looks healthy, and adding consumers does nothing, because a partition cannot be split. Classic causes: keying by country when 70 percent of traffic is one country, keying by a status field with three values, or keying by a null-heavy column so everything hashes to the same place.

The signal is simple and you can read it straight off the metadata: compare each topic's deepest partition against its average partition depth. A ratio far above 1 means the load is not spread.

Common mistake: treating log_end_offset as "how many messages are in this partition". It is a high-water mark, not a count. On a topic with short retention the two numbers differ by orders of magnitude, and a capacity plan built on the wrong one is wrong by that same factor.

Interview nuance: "how do you increase throughput on a topic?" has one honest answer and one trap. The honest answer is more partitions plus more consumers in the group. The trap is that adding partitions later rehashes keys onto different partitions, so records for a key that used to be in order can appear out of order relative to the older ones. Say that and you have shown you know partitions are a data-modeling decision, not a dial.

On a real platform this differs. Here you query one topic_partitions table. On a real cluster the same numbers come from kafka-topics --describe for the topic layout and kafka-run-class kafka.tools.GetOffsetShell (or a metrics pipeline) for the offsets, and on Amazon MSK they arrive as CloudWatch metrics per partition. Kafka 4.0, released in March 2025, removed ZooKeeper entirely: clusters run KRaft only, and controller failover dropped from tens of seconds to under five. Any tutorial that tells you to configure a ZooKeeper ensemble is pre-2025 and repeating it in an interview will date you. The vocabulary also travels: a Kinesis shard is a partition (sized for real in the closing lesson of this module), and a Microsoft Fabric Eventstream partition is the same idea again.

Sample data for this example

CREATE TABLE kafka_topics (
  topic              TEXT,
  partitions         INTEGER,
  replication_factor INTEGER,  -- copies of each partition across brokers
  retention_hours    INTEGER   -- how long a record survives before the broker deletes it from the head
);
INSERT INTO kafka_topics (topic, partitions, replication_factor, retention_hours) VALUES
  ('orders',            6, 3, 168),
  ('clickstream',      12, 3,  24),
  ('sessions',          4, 3,  72),
  ('payments',          3, 3, 720),
  ('inventory_updates', 1, 3, 168);
CREATE TABLE topic_partitions (
  topic            TEXT,
  partition_id     INTEGER,
  log_start_offset INTEGER,  -- oldest offset still retained; rises as retention deletes from the head
  log_end_offset   INTEGER,  -- the offset the NEXT produced record will receive
  leader_broker    INTEGER   -- the broker serving reads and writes for this partition
);
INSERT INTO topic_partitions (topic, partition_id, log_start_offset, log_end_offset, leader_broker) VALUES
  ('orders',             0,  200000,  640000, 1),
  ('orders',             1,  210000,  630000, 2),
  ('orders',             2,  190000,  670000, 3),
  ('orders',             3,  205000,  665000, 1),
  ('orders',             4, 3400000, 5000000, 2),
  ('orders',             5,  215000,  625000, 3),
  ('clickstream',        0, 5120000, 5900000, 1),
  ('clickstream',        1, 5050000, 5860000, 2),
  ('clickstream',        2, 4980000, 5800000, 3),
  ('clickstream',        3, 5200000, 5990000, 1),
  ('clickstream',        4, 15200000, 23400000, 2),
  ('clickstream',        5, 5010000, 5780000, 3),
  ('clickstream',        6, 5300000, 6140000, 1),
  ('clickstream',        7, 5150000, 5910000, 2),
  ('clickstream',        8, 5080000, 5930000, 3),
  ('clickstream',        9, 4900000, 5730000, 1),
  ('clickstream',       10, 5240000, 6040000, 2),
  ('clickstream',       11, 5110000, 6020000, 3),
  ('sessions',           0, 3100000, 4000000, 1),
  ('sessions',           1, 2600000, 3300000, 2),
  ('sessions',           2, 2400000, 3000000, 3),
  ('sessions',           3, 2450000, 3050000, 1),
  ('payments',           0,   40000,   92000, 1),
  ('payments',           1,   41000,   88000, 2),
  ('payments',           2,   39000,   84000, 3),
  ('inventory_updates',  0,    5000,  130000, 2);

Worked example (SQL)

-- The hot-partition check: how many messages each clickstream partition actually retains.
SELECT partition_id,
       log_start_offset,
       log_end_offset,
       log_end_offset - log_start_offset AS message_count
FROM topic_partitions
WHERE topic = 'clickstream'
ORDER BY message_count DESC;

Apply

Your turn

The task this lesson builds to.

Write a query that returns each partition of the clickstream topic with the number of messages it currently retains, as (partition_id, message_count), fullest first, over topic_partitions(topic, partition_id, log_start_offset, log_end_offset, leader_broker).

A partition retains everything between its start and end markers, so message_count is the difference between them and not the end marker alone. Alias the computed column exactly message_count.

3 hints and 1 automated check are waiting in the workspace.

Practice

Make it stick

A second problem on the same idea, plus 3 bonus drills.

Write a query that returns every topic whose fullest partition holds more than twice its average partition depth, as (topic, max_depth, avg_depth), deepest partition first, over the same topic_partitions table.

Depth is log_end_offset - log_start_offset. Compare each topic's maximum depth against its average depth and keep only the skewed topics. Alias the columns exactly max_depth and avg_depth.

3 hints and 1 automated check are waiting in the workspace.