Skip to main content
CodeSparring

Queue vs Pub/Sub vs Log/Streaming

Level 6: Level 6: Asynchronous & Event-Driven Systemsmedium30 minmessaging-modelskafkaqueue

A queue distributes work and deletes on ack (no replay); pub/sub fans a copy to every subscriber; a log retains an ordered stream that many consumer groups read at their own offset and can replay.

Three models hiding under "messaging"

Picking the wrong one is one of the most common design-review mistakes, so learn to name them precisely.

Point-to-point queue (Amazon SQS, RabbitMQ, a Redis list). Producers push messages; a pool of competing consumers pulls them. Each message is delivered to exactly one consumer in the pool, and once that consumer acks, the message is deleted. This is work distribution: add consumers to process faster, and no two workers do the same job. The defining property is that the message is consumed and gone. No replay, no second reader. RabbitMQ adds routing (exchanges, bindings) and per-message DLQs; SQS adds a visibility timeout so an un-acked message reappears for another worker after a crash.

Check yourself
Orders flow through an SQS queue that a worker pool drains. A second team asks to read the same messages for analytics. What do they find?

Pub/sub (Amazon SNS, Google Pub/Sub topics, RabbitMQ fanout). A producer publishes to a topic; every subscriber gets its own copy. This is fan-out: one OrderPlaced reaches email, analytics, and fraud independently. Classic pub/sub is often fire-and-forget: if a subscriber is down when the message is published and there is no per-subscriber durable queue, that subscriber misses it. The common production pattern is SNS-to-SQS fan-out, where the topic delivers a copy into each subscriber's own durable queue so slow or offline subscribers do not lose messages.

Log / stream (Apache Kafka, Amazon Kinesis, Apache Pulsar). Messages are appended to a durable, ordered, append-only log and retained by time or size (say seven days), regardless of who has read them. Consumers do not delete messages; each consumer group tracks its own offset (a cursor) into the log and reads forward. Because the data stays and each group has an independent cursor, a log gives you both fan-out (many groups) and replay (rewind an offset to reprocess history). A brand-new analytics team can start today and read the last 30 days from offset zero.

The two axes: retention and who tracks delivery

Table
The two axes that separate the models: whether consuming deletes the message, and who owns the position. Consumer-owned offsets are the whole reason a log gives replay and many independent reader groups.
ModelDelete on consume?Who tracks position?Replay?Fan-out?
Queue (SQS)yesbroker (per-message ack)nono (competing consumers)
Pub/Sub (SNS)yes (per subscriber)broker (per subscriber)noyes
Log (Kafka)noconsumer (own offset)yesyes (per group)
The two axes that separate the models: whether consuming deletes the message, and who owns the position. Consumer-owned offsets are the whole reason a log gives replay and many independent reader groups.

A queue's broker tracks per-message delivery and acks; it is push-ish and the broker owns state. A log is pull-based: the consumer owns its offset, which is why one slow consumer group cannot slow another and why replay is just "reset my offset." That consumer-owned-offset design is the whole reason a log scales to many independent readers and supports reprocessing.

Check yourself
On a Kafka topic, the analytics consumer group falls two days behind. What happens to the ranking group reading the same topic?

Interview nuance: the classic wrong turn is choosing a queue when the requirement is "multiple independent teams, each reading the full stream, some needing to replay 30 days." A queue deletes on consume and serves one consumer per message. The moment you hear "replay," "reprocess," or "N independent consumer groups over the same data," reach for a log.

Recap: a queue distributes work and deletes on ack (no replay); pub/sub fans a copy to every subscriber; a log retains an ordered stream that many consumer groups read at their own offset and can replay.

Check yourself

Name the messaging model each requirement points to.

Thumbnail jobs where each job must be processed by exactly one worker, then disappear
A new team must reprocess the last 30 days of the stream from offset zero
Fan a copy of each message into every subscriber's own durable queue, with no need for history
An un-acked message must reappear for another worker after a crash
Several independent reader groups over the same retained stream, each at its own cursor

Apply

Your turn

The task this lesson builds to.

Design the messaging backbone for an order system that needs (a) exactly one worker per order, (b) multiple independent subscribers to 'order placed', and (c) 30-day replay for a new analytics team; map each requirement to a model.

Think about

  1. How does retention differ between a queue and a log?
  2. Why does a log support many independent consumer groups and replay?
  3. What is the difference between broker-tracked delivery and consumer-driven offsets?

Practice

Make it stick

A second problem on the same idea, so it survives past today.

Design the event backbone for DoorDash-scale order events (roughly 100k orders per hour at peak) where the same OrderCreated stream must feed a real-time dispatch matcher, a per-restaurant notification service, a fraud pipeline, and a data-warehouse ingest that reprocesses the last 7 days whenever the ML team ships a new feature. Choose the model(s) and justify retention and consumer topology.

Think about

  1. Which two requirements structurally disqualify a queue as the backbone?
  2. What partition key preserves the ordering that actually matters?
  3. Who sets the retention window: the fastest consumer or the replay need?