Systems Analyst: What is CDC (Change Data Capture) and when to use it?

CDC (Change Data Capture) — capture and propagate data changes in real-time.

Methods:

• Log-based: Read database WAL/binlog (Debezium, Maxwell).

  • Minimal impact on source database.
  • Captures all changes including deletes.

• Trigger-based: Database triggers capture changes.

  • Higher overhead, more flexible.

• Polling: Periodically query for changes.

  • Simple but high latency and DB load.

Use cases:

• Data warehouse loading (real-time ETL).
• Microservice data synchronization.
• Event sourcing.
• Cache invalidation.

Architecture:

• Source DB → CDC tool → Kafka → Consumers.
• Consumers: data warehouse, search index, cache.

Tools:

• Debezium (open-source, Kafka Connect).
• AWS DMS.
• Flink CDC.

Challenges:

• Schema evolution handling.
• Ordering guarantees.
• Initial snapshot + ongoing changes.