Spring Boot and Observability (Metrics, Tracing, Logging)?

Question

Accepted Answer

**Observability** — the ability to understand what is happening inside an application through its **external manifestations** (metrics, tracing, logs).

**Simple analogy:** Observability is like an airplane's instrument panel: altitude (latency), speed (throughput), fuel (memory), pressure (CPU). The pilot doesn't look inside the engine — they look at the instruments. Without observability, you are flying blind. When something goes wrong at 3 AM, you need to know WHERE (tracing), HOW MUCH (metrics), and WHY (logs).

**Why it matters in production:** In a microservices architecture with 50 services, a single user request may flow through 10 services. Without observability, debugging "why is the user seeing a timeout?" requires logging into each server, grepping log files, and guessing. With observability, you trace the exact request across all services, see which one added 5 seconds of latency, and find the error log with full context.

**Three pillars:**

**1. Metrics (quantitative measurements over time):**
```yaml
management:
  metrics:
    export:
      prometheus:
        enabled: true
    distribution:
      percentiles-histogram:
        http.server.requests: true
      percentiles:
        http.server.requests: 0.5, 0.95, 0.99
```

```java
// Custom metrics
@Service
public class OrderService {
    private final Counter orderCounter;
    private final Timer orderTimer;
    private final DistributionSummary orderSize;

public OrderService(MeterRegistry registry) {
        this.orderCounter = Counter.builder("orders.created")
            .description("Total orders created")
            .tag("service", "order")
            .register(registry);
        this.orderTimer = Timer.builder("orders.processing")
            .publishPercentiles(0.5, 0.95, 0.99)
            .register(registry);
        this.orderSize = DistributionSummary.builder("orders.items")
            .register(registry);
    }

public Order create(Order order) {
        return orderTimer.record(() -> {
            Order result = process(order);
            orderCounter.increment();
            orderSize.record(order.getItems().size());
            return result;
        });
    }
}
```

**2. Tracing (following a single request across services):**
```yaml
management:
  tracing:
    sampling:
      probability: 1.0  # 100% in dev, 0.1 in prod to reduce overhead
  zipkin:
    tracing:
      endpoint: http://localhost:9411/api/v2/spans
```

```java
// Custom spans
@Service
public class PaymentService {
    private final ObservationRegistry observationRegistry;

public PaymentResult process(Payment payment) {
        return Observation.createNotStarted("payment.process", observationRegistry)
            .lowCardinalityKeyValue("method", payment.getMethod())
            .observe(() -> {
                // Your payment logic here
                return gateway.charge(payment);
            });
    }
}
```

**3. Logging (structured, contextual):**
```yaml
logging:
  structure:
    json:
      enabled: true  # Spring Boot 3.2+
  pattern:
    level: "%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}]"
```

```java
@Slf4j
@Service
public class OrderService {
    public Order create(Order order) {
        // traceId and spanId are automatically included via MDC
        log.info("Creating order for user {}", order.getUserId());
        log.debug("Order items: {}", order.getItems());
        try {
            return repository.save(order);
        } catch (Exception e) {
            log.error("Failed to create order for user {}", order.getUserId(), e);
            throw e;
        }
    }
}
```

**Production stack:**
```
Spring Boot Actuator → Micrometer → Prometheus → Grafana dashboards
Spring Boot Actuator → Micrometer Tracing → Zipkin / Jaeger for distributed tracing
SLF4J → Logback → ELK Stack (Elasticsearch + Logstash + Kibana) for log aggregation
Alerting: Grafana Alerts → PagerDuty / Slack notifications
```

**Common pitfalls:**
- **Too much logging in production** — `DEBUG` level creates gigabytes of logs. Use `INFO` for production.
- **High-cardinality metric labels** — `tag("userId", userId)` creates millions of time series. Use low-cardinality labels like `tag("status", "success")`.
- **Sampling at 100% in production** — tracing overhead adds up. Sample 1-10% of requests.
- **Missing health checks** — always expose `/actuator/health` for load balancers and Kubernetes probes.

Java: Spring Boot and Observability (Metrics, Tracing, Logging)?