What is the difference between Kafka and RabbitMQ?

Question

Accepted Answer

**Kafka** — a distributed event streaming platform designed for high-throughput, fault-tolerant, durable event streaming. **RabbitMQ** — a traditional message broker implementing AMQP, designed for flexible routing and message queuing. **System Design Context:** Kafka is optimized for throughput and event replay — it stores messages durably and allows multiple consumers to read the same events. RabbitMQ is optimized for message routing and delivery — it removes messages after consumption. **Kafka — Detailed Characteristics:** - **Distributed commit log** — messages stored in ordered, append-only partitions. - **Durable** — messages persisted to disk, replicated across brokers. - **Replay** — consumers can re-read events from any offset. - **High throughput** — millions of messages per second. - **Ordered within partition** — global order requires single partition. - **Consumer groups** — parallel consumption across instances. ```java // Kafka producer with headers and partitioning @Service public class EventProducer { private final KafkaTemplate kafkaTemplate; public void sendEvent(OrderEvent event) { ProducerRecord record = new ProducerRecord<>("order-events", event.orderId().toString(), event); record.headers().add("source", "order-service".getBytes()); record.headers().add("version", "1.0".getBytes()); kafkaTemplate.send(record); } } // Kafka consumer with manual offset management @Service public class EventConsumer { @KafkaListener( topics = "order-events", groupId = "analytics-service", containerFactory = "kafkaListenerContainerFactory" ) public void consume( @Payload OrderEvent event, Acknowledgment ack) { analyticsService.process(event); ack.acknowledge(); // manual commit } } ``` **RabbitMQ — Detailed Characteristics:** - **Traditional message broker** — messages removed after consumption. - **AMQP protocol** — flexible routing with exchanges and bindings. - **No replay** — once consumed, message is gone. - **Flexible routing** — direct, topic, fanout, headers exchanges. - **Message acknowledgment** — explicit ack/nack. - **Dead letter queues** — route failed messages for retry. ```java // RabbitMQ producer @Service public class TaskPublisher { private final RabbitTemplate rabbitTemplate; public void publishTask(Task task) { rabbitTemplate.convertAndSend( "task-exchange", "task.processing", task, message -> { message.getMessageProperties() .setHeader("priority", "high"); message.getMessageProperties() .setExpiration("60000"); // TTL: 60s return message; }); } } // RabbitMQ consumer with retry @Service public class TaskConsumer { @RabbitListener( queues = "task-processing-queue", containerFactory = "rabbitListenerContainerFactory" ) public void consume(Task task, Channel channel, @Header(AmqpHeaders.DELIVERY_TAG) long tag) { try { taskService.process(task); channel.basicAck(tag, false); } catch (Exception e) { channel.basicNack(tag, false, true); // requeue } } } ``` **Performance Comparison:** - Kafka: millions msg/sec, batch-oriented, high latency per message. - RabbitMQ: tens of thousands msg/sec, low latency, message-by-message. **Common Pitfalls:** - **Kafka**: consumer lag grows if processing is slow. Fix: scale consumers, optimize processing. - **RabbitMQ**: message loss if broker crashes before persistence. Fix: durable queues, persistent messages. **When to Use:** - **Kafka** — event sourcing, streaming, log aggregation, real-time analytics, audit trails. - **RabbitMQ** — task queues, RPC, complex routing, request-reply patterns.

Java: What is the difference between Kafka and RabbitMQ?