Kafka Architecture Internals

Not a fast queue: a replicated commit log

Kafka is a distributed, replicated, append-only commit log, and almost every property people admire falls out of that one design choice. A topic is a named log split into partitions. Each partition is an ordered, immutable sequence of records, and every record gets a monotonically increasing offset (0, 1, 2, ...). That is the entire data model: no per-message delete, no random insert, no in-place update. Producers append to the tail; consumers read forward from an offset they control.

Why it is fast

Sequential disk writes: appending to the end of a file is the one access pattern spinning disks and SSDs both love, so Kafka sustains hundreds of MB/s per broker. Page cache: Kafka writes to the OS page cache and lets the kernel flush, so recent data is served from RAM with no user-space copy. Zero-copy: on read, sendfile() moves bytes from page cache straight to the network socket without dragging them through the JVM heap. Add producer-side batching and compression (lz4/zstd, batches keyed by linger.ms and batch.size) and one cluster handles millions of events per second.

Durability from replication

Each partition has one leader and N-1 followers (replication factor typically 3). Followers pull from the leader and, when caught up, sit in the in-sync replica (ISR) set. Two settings decide the trade:

• acks on the producer: acks=0 (fire and forget, can lose data), acks=1 (leader persisted, but a leader crash before replication loses acknowledged writes), acks=all (leader waits for all ISR members).
• min.insync.replicas on the broker: the minimum ISR size for an acks=all write to be accepted. With RF=3 and min.insync.replicas=2, a write needs the leader plus one follower, so you survive one broker loss with zero acknowledged-message loss and still accept writes.

Interview nuance: acks=all alone is not durable. If min.insync.replicas=1, "all ISR" can mean "just the leader" after followers drop out, so a leader crash still loses acknowledged writes. The durable combination is acks=all and min.insync.replicas>=2 and RF>=3.

Topic "rides", partition 3:
 offset:  0    1    2    3    4  <- append here (tail)
 record: [r0] [r1] [r2] [r3] [r4]
 Leader (broker 1) --replicate--> Follower (b2), Follower (b3)
 ISR = {1,2,3}. acks=all + min.insync.replicas=2 -> survives 1 loss.

Log segments and retention: a partition is stored as segment files that roll by size/time; old segments are deleted (time/size retention) or compacted. Tiered storage (KIP-405) offloads cold segments to S3-class object storage so retention cost decouples from broker disk. Finally, KRaft (GA, default in Kafka 4.0) replaced ZooKeeper: cluster metadata now lives in an internal Raft quorum of controllers, removing the external dependency, speeding failover, and scaling to far more partitions.

Recap: Kafka is a partitioned append-only log; sequential writes, page cache, and zero-copy explain its throughput; durability is leader/follower replication tuned by acks plus min.insync.replicas over the ISR (durable = acks=all + min.insync.replicas>=2 + RF3); retention, segments, compaction, and tiered storage govern cost and replay; and KRaft removed ZooKeeper by making metadata a Raft quorum.