Ecommet
Topic 57 of 75

Logs, Metrics, Traces and SLOs

Instrumentation is how a system reports on itself. Learn when to use logs, metrics and traces, how request IDs tie them together, and how SLIs, SLOs, SLAs and error budgets turn "reliable enough" into numbers.

Intermediate 4 min read 4 quiz questions

Jump to the quiz

In a nutshell

  • Metrics: cheap numbers over time, for dashboards and alerts. Logs: detailed records of events. Traces: the path of one request across services.
  • Pass a trace ID through every call and include it in logs, so you can jump from a slow trace to its logs.
  • SLI: what you measure (share of fast, successful requests). SLO: the target (99.9% over 30 days). SLA: the contract, with penalties.
  • The error budget (100% − SLO) is how much unreliability you can spend on releases and experiments.

Three kinds of telemetry

MetricsLogsTraces
WhatNumbers aggregated over time: counters, gauges, histogramsTimestamped records of eventsA tree of timed spans for one request
AnswersIs something wrong? How much?What exactly happened here?Where did this request spend its time?
CostLow, fixed per seriesGrows with trafficGrows with traffic; usually sampled
Watch outHigh-cardinality labels (user ID) explode series countsUnstructured text is hard to search; never log secretsNeeds context passed through every call

OpenTelemetry is the vendor-neutral standard for producing all three: one set of SDKs and a common format, sent to whichever backend you use (Prometheus and Grafana, Jaeger, a commercial service).

A structured log line: searchable fields, and the trace ID that links it to the request

Example
{"ts":"2026-10-29T10:42:07.311Z","level":"error","service":"checkout",
 "trace_id":"4bf92f3577b34da6","order_id":1042,"msg":"payment declined","code":"card_expired"}

Distributed tracing

One request, one trace

One request, one trace Gateway → Checkout: trace 4bf9: span A Checkout → Database: span B: 4 ms Checkout → Payments: span C Payments → Payments: bank API: 1,800 ms Payments → Checkout Checkout → Gateway: total 1,850 ms GatewayCheckoutPaymentsDatabasetrace 4bf9: span Aspan B: 4 msspan Cbank API: 1,800 mstotal 1,850 ms
Every service adds a span with timings under the same trace ID, so the slow hop stands out.
Diagram as text
  • Gateway → Checkout: trace 4bf9: span A
  • Checkout → Database: span B: 4 ms
  • Checkout → Payments: span C
  • Payments → Payments: bank API: 1,800 ms
  • Payments → Checkout
  • Checkout → Gateway: total 1,850 ms

SLIs, SLOs and SLAs

TermMeansExample
SLI (indicator)A measured ratio of good events to all eventsRequests answered successfully in under 300 ms ÷ all requests
SLO (objective)The target for the SLI over a window99.9% over 30 days
SLA (agreement)A promise to customers with consequences99.5% monthly, or service credits
Error budget1 − SLO: allowed bad events0.1% of requests, about 43 minutes of full downtime per 30 days

The SLO is set tighter than the SLA, so you notice and react before customers are owed anything. And 100% is the wrong target: users cannot tell 99.99% from 100% through their own flaky networks, and chasing it stops all change.

Percentiles and the remaining error budget

Java
import java.util.*;

class Slo {
    // Nearest-rank percentile, p in (0, 100].
    static double percentile(List<Double> values, double p) {
        List<Double> sorted = new ArrayList<>(values);
        Collections.sort(sorted);
        int rank = (int) Math.ceil(p / 100.0 * sorted.size());
        return sorted.get(Math.max(rank, 1) - 1);
    }

    // Share of the error budget still unspent (negative means the SLO is broken).
    static double budgetRemaining(double slo, long total, long bad) {
        double allowed = (1 - slo) * total;
        return (allowed - bad) / allowed;
    }
}
Python
import math

def percentile(values, p):
    """Nearest-rank percentile, p in (0, 100]."""
    ordered = sorted(values)
    rank = math.ceil(p / 100 * len(ordered))
    return ordered[max(rank, 1) - 1]

def budget_remaining(slo, total, bad):
    """Share of the error budget still unspent (negative means the SLO is broken)."""
    allowed = (1 - slo) * total
    return (allowed - bad) / allowed

Common mistakes

  • Metric labels with unbounded values (user IDs, URLs with IDs), exploding storage and cost.
  • Free-text logs with no request or trace ID.
  • Logging passwords, tokens or personal data.
  • Setting SLOs at 100%, or not at all.

Interview questions

Try answering out loud before you open each one.

How would you define reliability for this API?

Pick SLIs users care about: availability (share of non-5xx responses) and latency (share under a threshold, say 300 ms), measured at the load balancer. Set SLOs such as 99.9% over 30 days for each. Alert on fast burn of the error budget, and use the remaining budget to decide release pace.

Check yourself

1. Which telemetry shows where one request spent its time across five services?
Show the answer

Answer: A distributed trace. Spans under one trace ID show each hop.

2. SLO 99.9% for 1,000,000 requests. How many may fail?
Show the answer

Answer: 1,000. 0.1% of a million.

3. How should an SLO relate to the SLA?
Show the answer

Answer: Tighter, so you react before breaking the contract. The gap is your safety margin.

4. Why avoid a user ID as a metric label?
Show the answer

Answer: Each value creates a new time series, exploding cardinality. Put per-user detail in logs or traces instead.

Further reading

Your notes

Saved in this browser as you type.

Finished this topic?

Mark it read to track your progress, then carry on with the next one.

Next topic