Observability: Micrometer، Prometheus و OpenTelemetryObservability: Micrometer, Prometheus & OpenTelemetry

It's 2 a.m. Your order service has gone "slow" — not down, just slow. Some users get errors, some don't. The code that worked yesterday is byte-for-byte the same today. Now answer this: where is the problem? The database? A downstream service? A queue that filled up? One user sending weird input? If your only tool is to SSH into a box and eyeball log files, you'll be there until dawn. A system you can understand from the outside — figuring out what's happening inside just by looking at what it emits — is a system that has observability. This chapter is about building exactly that capability.

This chapter is the operational heart of every serious distributed system. As always, we grab the concept with an analogy first, then learn its technical name, then wire it to real code. Don't fear the new words — I'll define each one the moment it first appears.

Part 0 — a few words you must "feel" before we start

Three words recur throughout this chapter. Let me plant them with analogies now so you don't get stuck later.

• Telemetry: "the data a system produces and sends out about itself." Its Greek roots are tele ("far") and metron ("measure") — literally "measuring from a distance." Every metric, log, and trace is a piece of telemetry.
• Instrumentation: "the work you do so your code produces telemetry" — like fitting sensors onto an engine. When you add a counter.increment() line, you are instrumenting your code.
• Cardinality: "the number of distinct values something can take." Gender has low cardinality (a couple of values); a user ID has very high cardinality (millions). This innocent-looking word will later star in the chapter's most dangerous trap.

Part 1 — The three pillars of observability

The information a system emits about itself comes in three different flavors, called the three pillars of observability. Each answers a different kind of question, and the classic beginner mistake is reaching for one pillar to answer another pillar's question.

Let's define them precisely:

For the rest of the chapter we focus mainly on metrics (that's where most of the Java ecosystem's tooling lives), then move to tracing and OpenTelemetry. Logging has its own dedicated chapter.

Part 2 — Micrometer: "SLF4J for metrics"

Say you've decided to collect metrics. But send them where? Prometheus? Datadog? CloudWatch? Graphite? New Relic? Each has its own API and format. If you code directly against one vendor's API, the day the company decides to migrate, you're touching a thousand places in your code.

If you've read the logging chapter this pattern is familiar: SLF4J was a facade over Logback and Log4j2. Micrometer plays exactly that role for metrics — a vendor-neutral facade. The Micrometer team itself introduces it with the phrase "SLF4J for metrics."

Two key Micrometer concepts:

• Meter: the unit of measurement — a counter, a gauge, a timer. Anything "you measure" is a meter.
• MeterRegistry: the factory and warehouse for all meters. You create meters from the registry, and the type of registry decides where output goes. A PrometheusMeterRegistry prepares output for Prometheus; a SimpleMeterRegistry just holds it in memory (great for tests).

In Spring Boot you almost never build these by hand — auto-configuration creates a global registry for you. Just add these dependencies:

<!-- Actuator exposes management endpoints, including metrics -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

<!-- This dependency makes the "output language" Prometheus -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

With just that second dependency present, Spring Boot exposes an /actuator/prometheus endpoint that Prometheus can read metrics from. If tomorrow you move to Datadog, you swap micrometer-registry-prometheus for micrometer-registry-datadog and your instrumentation code stays untouched. That's the whole value of a facade.

Part 3 — The four meter types you must know

Micrometer has several meters, but four are the main pillars. Grab their difference with an analogy, then see the code.

1) Counter — only goes up

A monotonically increasing counter; it never decreases, only rises or holds (until the app restarts). Use it to count "how many times something happened": request count, error count, orders placed.

import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;

public class OrderService {
    private final Counter ordersPlaced;

    public OrderService(MeterRegistry registry) {
        this.ordersPlaced = Counter.builder("orders.placed")   // metric name
                .description("Total number of orders placed")
                .tag("channel", "web")                          // a dimension
                .register(registry);
    }

    public void placeOrder(Order order) {
        // ... order-placing logic ...
        ordersPlaced.increment();   // go up by one
    }
}

2) Gauge — a needle that goes up and down

An instantaneous value that at any moment can be higher or lower: items in a queue, active DB connections, memory usage percent. Key point: you observe a gauge, you don't set its value by hand. You hand Micrometer an object and a function, and whenever Micrometer wants to read the metric, it calls that function to get the current value.

import io.micrometer.core.instrument.Gauge;
import java.util.Queue;

Queue<Task> pending = ...;

Gauge.builder("queue.size", pending, Queue::size)  // object + a function giving the value
        .description("Number of tasks waiting in the queue")
        .register(registry);

3) Timer — a stopwatch for short events

The single most important meter for performance monitoring. A Timer records two things at once: how many times the event happened and the sum of all durations. From these two, the query system can derive average, throughput, and — if a histogram is enabled — percentiles.

import io.micrometer.core.instrument.Timer;

Timer requestTimer = Timer.builder("http.server.requests")
        .description("Request processing time")
        .tag("uri", "/checkout")
        .publishPercentileHistogram()   // publish histogram buckets (server-side percentiles)
        .register(registry);

// Style 1: with a block of code
String result = requestTimer.record(() -> handleCheckout());

// Style 2: manual, with Timer.Sample when start and stop are separate
Timer.Sample sample = Timer.start(registry);
try {
    handleCheckout();
} finally {
    sample.stop(requestTimer);   // records duration from start to here
}

In Spring, instead of writing all this by hand, you can use the @Timed annotation to time a method automatically:

import io.micrometer.core.annotation.Timed;

@Timed(value = "checkout.duration", description = "Checkout process duration", percentiles = {0.5, 0.95, 0.99})
public Receipt checkout(Cart cart) {
    // ...
}

4) DistributionSummary — the distribution of sizes (not time)

Just like a Timer, but for measuring something whose unit isn't time: a request payload's size in bytes, the number of items in a cart, the size of a batch. Same distribution and percentile capabilities.

import io.micrometer.core.instrument.DistributionSummary;

DistributionSummary payloadSize = DistributionSummary.builder("http.request.size")
        .baseUnit("bytes")
        .publishPercentiles(0.5, 0.95, 0.99)   // compute client-side percentiles
        .register(registry);

payloadSize.record(requestBody.length);

Part 4 — Tags: from a flat metric to a multi-dimensional one

So far our metrics have been a single plain number. But the real power of modern systems is in dimensions. A tag (Prometheus calls it a label) is a key-value pair attached to a metric that lets you slice it later.

// One metric, many dimensions — same name with different tag combinations
registry.counter("http.server.requests",
        "method", request.getMethod(),      // GET / POST / ...
        "uri", request.getUri(),            // /checkout / /cart / ...
        "status", response.getStatus())     // 200 / 404 / 500 ...
    .increment();

Every unique combination of tag values creates a separate time series. So {method=GET, uri=/cart, status=200} is one series and {method=POST, uri=/cart, status=500} is another. And this is exactly the trap I warned about back in Part 0.

Part 5 — Prometheus: the pull model and building percentiles

Now we produce metrics; but who collects and stores them? The most common answer in the cloud-native world is Prometheus — an open-source time-series database developed under the CNCF umbrella.

The advantage of pull is that Prometheus itself knows what to scrape (service discovery), and if an app doesn't respond, that "not responding" is itself a health signal (that instance is "down"). A simple config:

# prometheus.yml — tells Prometheus where and how often to scrape
global:
  scrape_interval: 15s        # scrape every 15 seconds

scrape_configs:
  - job_name: 'order-service'
    metrics_path: '/actuator/prometheus'   # the endpoint Micrometer created
    static_configs:
      - targets: ['order-service:8080']

When Prometheus reads that endpoint, it sees plain text (called the exposition format), something like:

# HELP http_server_requests_seconds  Request processing time
# TYPE http_server_requests_seconds histogram
http_server_requests_seconds_bucket{uri="/checkout",le="0.05"} 24
http_server_requests_seconds_bucket{uri="/checkout",le="0.1"}  57
http_server_requests_seconds_bucket{uri="/checkout",le="0.5"}  92
http_server_requests_seconds_bucket{uri="/checkout",le="+Inf"} 100
http_server_requests_seconds_count{uri="/checkout"} 100
http_server_requests_seconds_sum{uri="/checkout"}   12.4

Histograms: how we build percentiles from individual timings

Here you must understand Prometheus's most important technical idea: how can you compute the 99th percentile without keeping every individual measurement?

Notice the le in the output above: short for less than or equal to. Each line _bucket{le="0.1"} 57 means "57 requests had a duration less than or equal to 0.1 seconds." These are cumulative: a bigger bucket includes all smaller ones too. Now the query system can estimate a percentile by interpolating between buckets. In Prometheus that's done by the histogram_quantile function:

# 95th percentile of request duration over the last 5 minutes
histogram_quantile(0.95, rate(http_server_requests_seconds_bucket[5m]))

Part 6 — PromQL and Grafana: from numbers to decisions

Raw data alone is useless; you must be able to ask questions of it. Prometheus's query language is PromQL. A few frequent examples you should have at your fingertips:

# Requests per second (RED: Rate) — derived from a counter
sum(rate(http_server_requests_seconds_count[5m])) by (uri)

# Error percentage (RED: Errors) — ratio of 5xx to total
sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
  /
sum(rate(http_server_requests_seconds_count[5m]))

# 99th percentile latency (RED: Duration)
histogram_quantile(0.99, sum(rate(http_server_requests_seconds_bucket[5m])) by (le))

Notice rate(...[5m]): because a counter only goes up, its raw value is meaningless; rate takes its derivative over the 5-minute window and turns it into "per second." This is exactly what I said in the Counter part: store raw, compute the rate at query time.

Part 7 — Distributed tracing: following one request's trail

Metrics tell you "5% of requests are slow," but in a microservice architecture where a single user request passes through ten different services, a metric doesn't tell you which service and which step is to blame. Here the third pillar enters: distributed tracing.

Let's sharpen the words:

• Span: a named unit of work with a start and end time — e.g. "database query" or "call the payment service." Each span can have attributes (descriptive key-values), events (point-in-time moments within it), and a status (ok/error).
• Trace: the collection of all spans of one request, connected as a tree of parent-child. The root of the tree is the first span (e.g. the request entering the API gateway).
• Trace context / propagation: so that spans from different services know they all belong to one trace, each service passes a shared identifier in an HTTP header when calling the next service. The industry standard for this is W3C Trace Context, which uses a header called traceparent.

The traceparent header looks like this and has a simple but precise structure:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^  ^                                ^                ^
          version  trace-id (16 bytes)          span-id (8 bytes)  flags

Part 8 — OpenTelemetry: the industry's single standard

Until now each signal had its own tool and standard. Historically this was chaos: Zipkin and Jaeger and OpenTracing and OpenCensus each had their own API. The problem: if you instrumented your code with one vendor's API, you were locked to it (vendor lock-in).

OpenTelemetry is a project under the CNCF umbrella (the result of merging OpenTracing and OpenCensus) and has become the industry's de-facto standard today. Its key components:

• API: the interface your code instruments against (creates spans, records metrics). Lightweight, no heavy dependencies.
• SDK: the actual implementation of the API — sampling, processing, and exporting.
• OTLP (OpenTelemetry Protocol): the single protocol that sends telemetry from your app to a back-end. Everyone's shared language.
• Collector: an intermediary service that receives telemetry from apps, processes/filters/enriches it, and forwards it to one or more back-ends. Like a central hub.
• Semantic Conventions: a standard agreement on names. For example, everyone agreed the HTTP method is stored in an attribute called http.request.method. This means dashboards and tools become portable across organizations.

Two ways to instrument with OTel in Java

Way 1 — the Java agent (no code changes): OpenTelemetry ships a Java agent that attaches to the JVM at runtime and automatically instruments dozens of common libraries (Spring, JDBC, Kafka, HTTP clients...). Just add a flag to the run:

java -javaagent:opentelemetry-javaagent.jar \
     -Dotel.service.name=order-service \
     -Dotel.exporter.otlp.endpoint=http://collector:4317 \
     -jar order-service.jar

Way 2 — manual instrumentation with the API: when you want custom spans for your own business logic:

import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;

Tracer tracer = openTelemetry.getTracer("order-service");

Span span = tracer.spanBuilder("validate-inventory").startSpan();
try (var scope = span.makeCurrent()) {
    span.setAttribute("order.id", orderId);      // descriptive attribute
    boolean ok = inventory.check(orderId);
    span.setAttribute("inventory.available", ok);
} catch (Exception e) {
    span.recordException(e);                      // record the error on the span
    throw e;
} finally {
    span.end();                                   // without this, the span never closes!
}

The bridge between Micrometer and OpenTelemetry: the Observation API

Here the beauty of Spring Boot 3's architecture becomes clear. Micrometer introduced an Observation API — an instrumentation layer you write once that simultaneously produces metrics and traces (and helps with log correlation).

import io.micrometer.observation.Observation;
import io.micrometer.observation.ObservationRegistry;

Observation.createNotStarted("order.checkout", observationRegistry)
    .lowCardinalityKeyValue("channel", "web")    // goes to the metric tag too (low-cardinality)
    .highCardinalityKeyValue("orderId", orderId) // only a span attribute (high-cardinality)
    .observe(() -> processCheckout(order));

Part 9 — What should you measure? RED, USE, and the Golden Signals

Instrumenting is easy; knowing what's worth measuring is the art. There are three standard industry frameworks you should know.

• RED is great for anything that takes a "request." Three simple numbers that reveal 90% of an API's health. Those same three PromQL queries in Part 6 were exactly RED.
• USE is for resources, not services. For each resource ask: how busy is it (utilization)? How much work is waiting (saturation, like queue length)? How many errors did it produce?
• Four Golden Signals is Google's version; effectively a blend of RED and USE with an emphasis on latency.

Part 10 — SLI, SLO, and SLA: the language of reliability

The last piece of the puzzle: how to talk about "how reliable should the system be" in a numeric, agreed-upon way. Three closely related but distinct acronyms that constantly get muddled.

Let's define them precisely:

• SLI (Service Level Indicator): a measured number about service quality. Example: "percent of requests answered in under 300ms with no error." This is what you actually measure — and, as you saw, it comes from metrics.
• SLO (Service Level Objective): an internal target on that SLI. Example: "99.9% of requests must succeed, over a 30-day window." This is the dev team's red line.
• SLA (Service Level Agreement): a formal contract with a customer that, if breached, has financial/legal consequences (like refunds). An SLA is always looser than the SLO so you keep a safety margin.

Part 11 — Pitfalls and best practices

Let me gather the top traps in one place — the ones that bite in interviews or real code:

• Cardinality explosion: never put userId, email, a raw URL, or a free-form error message as a metric tag. Put those in logs/traces. Tags are only for bounded, countable values.
• Average instead of percentile: the average latency hides slow users. Take p95/p99.
• Pre-computed rate: store the counter raw and derive the rate with rate() at query time; don't build a rate yourself.
• Summary in a distributed system: client-side percentiles (summary) aren't additive; for multiple instances use histogram.
• A span that never closes: forgetting span.end() in finally leaves the span open forever and corrupts the trace.
• Breaking the propagation chain: if one service doesn't pass the traceparent header, the trace is left incomplete.
• Gauge on a temporary object: the gauge's weak reference means if the object is garbage-collected, the gauge dies and returns NaN.
• Vendor lock-in: don't code directly against one vendor's API; use the Micrometer/OpenTelemetry facade so migration stays possible.

Best practices:

• Instrument your code once with OpenTelemetry or the Micrometer Observation API to get both metrics and traces.
• Name metrics per OTel's semantic conventions so dashboards are portable.
• Weave the three pillars: alert from a metric, locate with a trace, see details in a log.
• Define SLOs and monitor the error budget, not just "server up/down."
• For a new project, put OTLP and a Collector in place from day one so the back-end is swappable.

Interview Questions

Now it's time to test yourself. Answer each question yourself first, then open the answer.