Kafka or RabbitMQ: which message broker to choose

RabbitMQ is a classic message broker: it delivers tasks and commands to specific consumers with flexible routing and deletes messages once processed. Kafka is a distributed event log that keeps events for a set period so many consumers can read and replay them. Job queues usually need RabbitMQ; event streams and real-time analytics call for Kafka.

Here is how they differ in design and how to choose.

How does RabbitMQ differ from Kafka in design?

RabbitMQ works like a post office: a producer sends a message to an exchange, which routes it into queues by rules; a consumer takes the message, acknowledges it and it is deleted. Kafka works like a log: events are appended to the end of a topic and kept for a configured time, and each consumer tracks how far it has read.

The main differences follow from that:

• in RabbitMQ a message is usually handled by one consumer from a queue; in Kafka one event can be read independently by many consumer groups;
• in RabbitMQ processed messages disappear; in Kafka events can be re-read from any point within the retention period;
• RabbitMQ offers flexible routing by keys and patterns; Kafka scales through topic partitions;
• Kafka is built for very large event streams; RabbitMQ for reliable task delivery with acknowledgements.

When should you choose RabbitMQ?

Choose RabbitMQ when tasks or commands must be delivered reliably to workers: background jobs, communication between a few services, delayed messages, priorities. It is simpler to operate for small and medium systems and well supported by frameworks.

Typical scenarios:

1. Background jobs enqueued by one application and executed by workers on another stack.
2. Commands between services: "charge the payment", "send a notification", "generate a document".
3. Routing by type: orders from different regions go to different processing queues.
4. Delayed retries and a dead-letter queue for manual review.

If queues are needed within one application, Redis or the database is often enough — see job queues in web applications.

When should you choose Kafka?

Choose Kafka when events are a data stream many systems need — analytics, search, notifications, audit — and it matters to store and replay them. It handles very large volumes and lets you add a new consumer that reads the history from the beginning.

Typical scenarios:

• a stream of user events for analytics and recommendations;
• propagating changes from the main database to other systems — for example, into the analytics store we described in ClickHouse for analytics;
• an audit log that must not be lost and has to be retained;
• event exchange between many teams and services in a large system.

Photo: Unsplash

How do they compare on the key criteria?

On the key criteria, RabbitMQ wins on simplicity, routing and delivering individual tasks; Kafka wins on throughput, retention and replay. The choice is not about "which is faster" but about which of these properties you need.

Criterion	RabbitMQ	Kafka
Model	Message queues with acknowledgements	An event log with retention
After processing	The message is deleted	The event stays until retention expires
Several independent consumers	Via separate queues	Native, via consumer groups
Replaying history	No	Yes, from any position
Routing	Flexible: keys, patterns, headers	By topics and partitions
Throughput	High	Very high
Operational complexity	Lower	Higher, especially without a managed service
Typical use	Tasks and commands	Event streams and data

The designs in a nutshell

RabbitMQ:  order created → exchange → queue "emails"    → email worker (message deleted)
                                    → queue "warehouse" → warehouse worker (message deleted)

Kafka:     topic "orders": [e1][e2][e3][e4][e5] ...  kept for 7 days
             ├─ group "analytics" reads from e1
             ├─ group "search" has read up to e4
             └─ new group "recommendations" starts at e1

Do you need a broker at all?

Not every project needs a broker: with one application and background jobs inside it, a separate RabbitMQ or Kafka adds operational cost without clear benefit. A broker is justified once several services exchange messages or an event stream is needed by different consumers.

Signs it is time:

• several services on different stacks exchange tasks;
• several independent systems must receive the same event;
• you need guaranteed delivery with acknowledgements and retries;
• event volume has outgrown Redis or the database.

This is the same kind of decision as monolith versus services — see monolith vs microservices. In the cloud both systems are available as managed services, which cuts operational complexity considerably.

In short

• RabbitMQ is a broker for delivering tasks and commands with flexible routing; messages are deleted after processing.
• Kafka is an event log with retention and replay for many consumers.
• Background jobs and inter-service commands usually need only RabbitMQ; data streams need Kafka.
• Within one application, Redis or the database is often enough without a separate broker.
• Managed cloud versions remove most of the operational complexity.