DevOps: What is Incident Management in SRE?

Incident Management — The process of detecting, responding to, and resolving incidents.

Lifecycle:

1. Detection — monitoring detects a problem.
2. Triage — severity assessment (SEV1-SEV4).
3. Response — on-call engineer responds.
4. Mitigation — temporary fix.
5. Resolution — permanent fix.
6. Post-Mortem — root cause analysis.

SEV levels:

• SEV1: Complete outage. Response time: 5 min.
• SEV2: Partial outage. Response time: 15 min.
• SEV3: Degraded performance. Response time: 1 hour.
• SEV4: Minor issue. Response time: 4 hours.

Runbook — instructions for typical incidents:

1## Pod CrashLoopBackOff
21. `kubectl logs <pod> --previous`
32. `kubectl describe pod <pod>` (check OOM)
43. If OOM — increase limits.memory

Post-Mortem (blameless):

1. What happened.
2. Impact.
3. Root cause.
4. What was done.
5. Action items.

MTTD, MTTR: MTTD < 5 min, MTTR < 30 min for SEV1.