What are monitoring and alerting?

Question

Accepted Answer

**Monitoring** — observation.
**Alerting** — notification.

```yaml
# Prometheus Alert Rules
groups:
  - name: app
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value }}"

- alert: HighLatency
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
        for: 5m
        labels:
          severity: warning
          annotations:
            summary: "High latency detected"

- alert: PodRestartLoop
        expr: increase(kube_pod_container_status_restarts_total[1h]) > 5
        labels:
          severity: critical
```

```yaml
# AlertManager Config
global:
  resolve_timeout: 5m

route:
  receiver: slack
  group_by: [alertname, severity]
  group_wait: 30s
  repeat_interval: 1h

receivers:
  - name: slack
    slack_configs:
      - api_url: "https://hooks.slack.com/services/..."
        channel: "#alerts"
        title: "{{ .GroupLabels.alertname }}"
        text: "{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}"

- name: pagerduty
    pagerduty_configs:
      - service_key: "..." 
```

**Metrics (RED/USE):**
- Rate: requests/sec.
- Errors: errors/sec.
- Duration: latency.
- Utilization: resource load.

DevOps: What are monitoring and alerting?