Systems Analyst: How do you define SLAs for data pipelines?

Data SLA (Service Level Agreement) — contractual guarantees for data delivery.

SLA components:

• Freshness: Data available by X time (e.g., 6 AM daily).
• Completeness: Minimum % of records processed.
• Accuracy: Maximum error rate.
• Availability: Pipeline uptime percentage.

Defining SLAs:

• Understand business requirements (when is data needed?).
• Assess technical feasibility (processing time, dependencies).
• Add buffer for upstream delays.
• Define escalation procedures for breaches.

Monitoring:

• Data freshness checks (last partition timestamp).
• Row count validation (expected vs actual).
• Pipeline duration tracking.
• Alerting on SLA breach risk.

Tools:

• Airflow SLA sensors and callbacks.
• dbt source freshness.
• Great Expectations validations.

Example SLA:

• Daily report available by 06:00 UTC.
• 99.9% uptime.
• < 0.1% data loss.