Spring Boot and Apache Kafka?

Question

Accepted Answer

**Apache Kafka** — a distributed event streaming system. Spring Boot connects via **spring-kafka**.

**Simple analogy:** Kafka is like a post office with mailboxes (topics). You put a letter (message) in the "Orders" mailbox, and everyone who needs it (consumers) can pick up a copy. Letters are not lost — Kafka stores them on disk for a configurable retention period (days, weeks, or forever). Unlike a real post office, Kafka can handle millions of letters per second and multiple recipients can each read the same letter independently.

**Why it matters in production:** Kafka decouples services — the order service does not need to know about the notification service or the analytics service. It publishes an "OrderCreated" event, and any number of downstream services consume it independently. This enables event sourcing, audit trails, and replay of historical data — capabilities impossible with synchronous REST calls.

**Configuration:**
```yaml
spring:
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      group-id: my-group
      auto-offset-reset: earliest
      max-poll-records: 500
      enable-auto-commit: false
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      acks: all
      retries: 3
      properties:
        max.block.ms: 5000
        linger.ms: 20
        batch.size: 16384
```

**Producer (sending):**
```java
@Service
@Slf4j
public class OrderEventPublisher {
    @Autowired private KafkaTemplate<String, OrderEvent> kafkaTemplate;

public void publishOrderCreated(OrderEvent event) {
        kafkaTemplate.send("orders", event.getOrderId(), event)
            .addCallback(
                result -> log.info("Sent offset: {}", result.getRecordMetadata().offset()),
                ex -> log.error("Failed to send", ex)
            );
    }

@Transactional
    public void publishInTransaction(OrderEvent event) {
        // Kafka send participates in the DB transaction
        kafkaTemplate.executeInTransaction(kafka ->
            kafka.send("orders", event.getOrderId(), event)
        );
    }
}
```

**Consumer (receiving):**
```java
@Service
@Slf4j
public class OrderEventHandler {
    @KafkaListener(topics = "orders", groupId = "order-processor")
    public void handleOrderCreated(@Payload OrderEvent event,
            Acknowledgment ack, @Header(KafkaHeaders.RECEIVED_PARTITION) int partition) {
        log.info("Received order on partition {}: {}", partition, event.getOrderId());
        try {
            processOrder(event);
            ack.acknowledge();  // Manual acknowledgment
        } catch (Exception e) {
            log.error("Processing failed, will retry", e);
            // Nack to trigger retry / DLQ
        }
    }

@RetryableTopic(attempts = "3", backoff = @Backoff(delay = 1000))
    @KafkaListener(topics = "orders.DLT")  // Dead Letter Topic
    public void handleFailed(OrderEvent event) {
        log.error("Permanent failure for: {}", event.getOrderId());
        alertService.notify(event);
    }
}
```

**Production best practices:**
- **Partition count** — set via `NewTopic` bean: more partitions = more parallelism, but also more overhead.
- **Consumer groups** — each group gets its own offset tracking. Different groups = independent consumption.
- **Idempotent consumers** — Kafka guarantees at-least-once delivery. Your consumer must handle duplicates.
- **Monitoring** — use `kafka/consumer-group` metrics to track lag per consumer group.

**Kafka vs RabbitMQ:**
- Kafka — high throughput, replay, event sourcing, log aggregation.
- RabbitMQ — complex routing, guaranteed delivery, task queuing.
- Kafka retains messages; RabbitMQ deletes after consumption.

Java: Spring Boot and Apache Kafka?