How to correlate traces, metrics and logs?

Question

Accepted Answer

**Correlation** — linking the three pillars of observability through shared identifiers.

**Trace ID as an anchor:**
1. **Trace** → contains `trace_id`.
2. **Metrics** → add `trace_id` as label.
3. **Logs** → add `trace_id` to structured log.

**OpenTelemetry (automatic correlation):**
```typescript
import { trace, context, SpanStatusCode } from "@opentelemetry/api";

const tracer = trace.getTracer("my-app");

async function handleRequest(req) {
  return tracer.startActiveSpan("handleRequest", async (span) => {
    const traceId = span.spanContext().traceId;
    logger.info({ traceId, message: "Processing request" });
    
    try {
      const result = await callService(req);
      span.setStatus({ code: SpanStatusCode.OK });
      return result;
    } catch (err) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      span.recordException(err);
      throw err;
    } finally {
      span.end();
    }
  });
}
```

**Loki + Tempo (Grafana stack):**
```yaml
# Structured log with trace_id
{
  "level": "info",
  "message": "Request processed",
  "traceID": "abc123",
  "spanID": "def456",
  "service": "orders-api",
  "duration_ms": 42
}
```

**Grafana Explore:**
- Tempo → click on trace → "View logs" → filter by `trace_id`.
- Logs → click on `trace_id` → "View trace" → see the full path.

**Exemplars in Prometheus:**
```promql
http_request_duration_seconds_bucket{le="0.1"} # histogram
# Exemplar: {trace_id="abc123"} attached to bucket
```

**Summary:** Trace ID → single key for navigating between traces → logs → metrics.

DevOps: How to correlate traces, metrics and logs?