How do you design a real-time analytics system?

Question

Accepted Answer

**Real-time analytics** — process and visualize data as it arrives.

**Architecture:**
- **Ingestion:** Kafka, Kinesis, Pub/Sub.
- **Processing:** Apache Flink, Spark Streaming, ksqlDB.
- **Storage:** ClickHouse, Druid, Apache Pinot.
- **Visualization:** Grafana, Metabase, custom dashboards.

**Processing patterns:**
- **Stream processing:** Windowed aggregations, joins.
- **Complex event processing (CEP):** Pattern detection.
- **Lambda architecture:** Batch + stream layers.
- **Kappa architecture:** Stream-only processing.

**Key decisions:**
- **Latency:** Seconds vs minutes vs hours.
- **Throughput:** Events per second.
- **Consistency:** Exact-once vs at-least-once.
- **Windowing:** Tumbling, sliding, session.

**Common use cases:**
- Fraud detection.
- IoT monitoring.
- Real-time dashboards.
- Recommendation engines.

**Challenges:**
- State management.
- Late arriving data.
- Schema evolution.

Systems Analyst: How do you design a real-time analytics system?