How do you design data retention and archival policies?

Question

Accepted Answer

**Data retention** — how long data is kept before deletion or archival.

**Drivers:**
- **Regulatory:** GDPR (right to erasure), HIPAA, SOX.
- **Business:** Data useful for analytics, auditing.
- **Technical:** Storage costs, performance.

**Retention policy components:**
- **Data classification:** PII, financial, operational.
- **Retention periods:** Per data type (e.g., 7 years for financial).
- **Deletion method:** Hard delete, anonymization.
- **Archive tier:** Hot → Warm → Cold → Deleted.

**Implementation:**
- **Database:** TTL, partition expiry, scheduled jobs.
- **Object storage:** Lifecycle policies (S3, GCS).
- **Data warehouse:** Time-travel, clone, drop.

**Archive strategies:**
- Compressed backups to cold storage.
- Parquet files in S3 Glacier.
- Tape for compliance (rare).

**Monitoring:**
- Track storage growth.
- Alert on retention policy violations.
- Regular audits of archived data.

Systems Analyst: How do you design data retention and archival policies?