How to implement Disaster Recovery for Kubernetes with Velero and cross-cluster restore?

Question

Accepted Answer

**Cross-cluster DR with Velero** — restore to another cluster/region.

**Architecture:**
- Primary cluster → Velero backup → S3 bucket.
- DR cluster → Velero restore from S3.

**Velero setup (primary):**
```bash
velero install \
  --provider aws \
  --bucket my-velero-backups \
  --backup-location-config region=us-east-1 \
  --snapshot-location-config region=us-east-1 \
  --use-node-agent
```

**Automatic backup:**
```bash
velero schedule create daily-full \
  --schedule="0 2 * * *" \
  --ttl=720h \
  --include-namespaces production \
  --snapshot-volumes=true

velero schedule create hourly-incremental \
  --schedule="0 * * * *" \
  --ttl=168h \
  --include-namespaces production
```

**Restore to DR cluster:**
```bash
# Install Velero on DR cluster
velero install \
  --provider aws \
  --bucket my-velero-backups \
  --backup-location-config region=us-east-1 \
  --use-node-agent

# List backups
velero backup get

# Restore
velero restore create dr-restore-$(date +%Y%m%d) \
  --from-backup daily-full-20240115020000 \
  --namespace-mappings production:production-dr \
  --restore-volumes=true

# Monitoring
velero restore get
velero restore logs dr-restore-20240115
```

**Post-restore:**
```bash
# Update DNS on DR
aws route53 change-resource-record-sets \
  --hosted-zone-id Z123 \
  --change-batch file://dns-failover.json

# Verification
kubectl get pods -n production-dr
kubectl get svc -n production-dr
curl -H "Host: myapp.com" http://dr-endpoint/myapp.com/health
```

**Velero Plugins:**
- **CSI Plugin** — PV snapshots via CSI.
- **AWS Plugin** — EBS snapshots.
- **Restic** — file-level backup (deprecated).

**DR Testing (monthly):**
```bash
# 1. Restore to test namespace
velero restore create dr-test \
  --from-backup daily-full-latest \
  --namespace-mappings production:dr-test

# 2. Verify
kubectl get pods -n dr-test
kubectl exec -n dr-test deploy/myapp -- curl localhost:8080/health

# 3. Cleanup
kubectl delete namespace dr-test
```

**RPO/RTO:** RPO = frequency of backups (hourly). RTO = time to restore + DNS propagation (30-60 min).

DevOps: How to implement Disaster Recovery for Kubernetes with Velero and cross-cluster restore?