Broker Technology Selection

"We'll use Kafka" is usually the wrong reflex

The senior move is to name the decision drivers, then match the workload to the cheapest tool that satisfies them. Kafka is a superb distributed log, but it is also operationally heavy (partitions, consumer groups, rebalancing, retention tuning, and a ZooKeeper or KRaft quorum to run). If you do not need what it gives, you are paying its tax for nothing.

The drivers to reason about out loud:

• Throughput. Millions of messages/sec favors a partitioned log (Kafka, Kinesis, Pulsar). Thousands/sec is comfortable for any queue.
• Ordering. Need per-key ordering? A log gives per-partition order; SQS FIFO gives per-group order; standard SQS gives none.
• Retention and replay. Need to reprocess history or feed many independent consumers? You need a log. Queues delete on consume.
• Delivery guarantee. At-least-once is the default everywhere; per-message ack and redelivery are a queue strength; exactly-once-ish processing needs extra machinery.
• Routing complexity. Rich topic/header routing, priorities, per-message TTL, and DLQs are RabbitMQ's home turf.
• Ops budget. A small team with no streaming platform should lean on managed services (SQS, SNS, Google Pub/Sub, Kinesis, MSK) before self-hosting Kafka.

The landscape

Logs / streams:   Kafka, Pulsar, Kinesis   -> high throughput, ordering, retention, replay
Queues:           RabbitMQ, SQS            -> per-message ack, routing, DLQ, work distribution
Managed fan-out:  SNS, Google Pub/Sub      -> topic fan-out without running a broker
Ordered managed:  SQS FIFO                 -> per-group ordering, exactly-once-ish, lower throughput
Lightweight:      NATS, Redis Streams      -> low latency, simple ops, smaller durability guarantees

RabbitMQ is a smart broker for complex routing and per-message workflows at moderate scale; SQS is a zero-ops managed queue for work distribution and decoupling on AWS; Kafka is a durable replayable log for high-throughput streaming and multi-consumer fan-out.

Pulsar is the classic "why not Kafka" foil: it separates compute (brokers) from storage (BookKeeper), so you scale serving and storage independently, and it has first-class multi-tenancy, geo-replication, and tiered storage built in, supporting both queue and log semantics in one system. The cost is a more complex deployment. Choose it when multi-tenancy or independent compute/storage scaling is a real requirement, not by default. NATS and Redis Streams cover the low-latency, lightweight end when you want simple pub/sub or a small stream with minimal ops.

Interview nuance: the strongest answer is sometimes "no broker at all." If the requirement is a strong-consistency CRUD read after write, a broker adds latency and a stale-read window for nothing; a direct synchronous call or a database is correct. Reaching for Kafka to decouple two services that make ten calls a second is over-engineering you should call out.

Recap: match the broker to the drivers (throughput, ordering, retention/replay, delivery, routing, ops budget); use a log only when replay/throughput justify its ops, a queue for work distribution and routing, managed services when the team is small, and sometimes no broker at all.