Logs, Metrics, Traces and SLOs
Instrumentation is how a system reports on itself. Learn when to use logs, metrics and traces, how request IDs tie them together, and how SLIs, SLOs, SLAs and error budgets turn "reliable enough" into numbers.
In a nutshell
- Metrics: cheap numbers over time, for dashboards and alerts. Logs: detailed records of events. Traces: the path of one request across services.
- Pass a trace ID through every call and include it in logs, so you can jump from a slow trace to its logs.
- SLI: what you measure (share of fast, successful requests). SLO: the target (99.9% over 30 days). SLA: the contract, with penalties.
- The error budget (100% − SLO) is how much unreliability you can spend on releases and experiments.
Three kinds of telemetry
| Metrics | Logs | Traces | |
|---|---|---|---|
| What | Numbers aggregated over time: counters, gauges, histograms | Timestamped records of events | A tree of timed spans for one request |
| Answers | Is something wrong? How much? | What exactly happened here? | Where did this request spend its time? |
| Cost | Low, fixed per series | Grows with traffic | Grows with traffic; usually sampled |
| Watch out | High-cardinality labels (user ID) explode series counts | Unstructured text is hard to search; never log secrets | Needs context passed through every call |
OpenTelemetry is the vendor-neutral standard for producing all three: one set of SDKs and a common format, sent to whichever backend you use (Prometheus and Grafana, Jaeger, a commercial service).
A structured log line: searchable fields, and the trace ID that links it to the request
{"ts":"2026-10-29T10:42:07.311Z","level":"error","service":"checkout",
"trace_id":"4bf92f3577b34da6","order_id":1042,"msg":"payment declined","code":"card_expired"}Distributed tracing
One request, one trace
Diagram as text
- Gateway → Checkout: trace 4bf9: span A
- Checkout → Database: span B: 4 ms
- Checkout → Payments: span C
- Payments → Payments: bank API: 1,800 ms
- Payments → Checkout
- Checkout → Gateway: total 1,850 ms
SLIs, SLOs and SLAs
| Term | Means | Example |
|---|---|---|
| SLI (indicator) | A measured ratio of good events to all events | Requests answered successfully in under 300 ms ÷ all requests |
| SLO (objective) | The target for the SLI over a window | 99.9% over 30 days |
| SLA (agreement) | A promise to customers with consequences | 99.5% monthly, or service credits |
| Error budget | 1 − SLO: allowed bad events | 0.1% of requests, about 43 minutes of full downtime per 30 days |
The SLO is set tighter than the SLA, so you notice and react before customers are owed anything. And 100% is the wrong target: users cannot tell 99.99% from 100% through their own flaky networks, and chasing it stops all change.
Percentiles and the remaining error budget
import java.util.*;
class Slo {
// Nearest-rank percentile, p in (0, 100].
static double percentile(List<Double> values, double p) {
List<Double> sorted = new ArrayList<>(values);
Collections.sort(sorted);
int rank = (int) Math.ceil(p / 100.0 * sorted.size());
return sorted.get(Math.max(rank, 1) - 1);
}
// Share of the error budget still unspent (negative means the SLO is broken).
static double budgetRemaining(double slo, long total, long bad) {
double allowed = (1 - slo) * total;
return (allowed - bad) / allowed;
}
}import math
def percentile(values, p):
"""Nearest-rank percentile, p in (0, 100]."""
ordered = sorted(values)
rank = math.ceil(p / 100 * len(ordered))
return ordered[max(rank, 1) - 1]
def budget_remaining(slo, total, bad):
"""Share of the error budget still unspent (negative means the SLO is broken)."""
allowed = (1 - slo) * total
return (allowed - bad) / allowedCommon mistakes
- Metric labels with unbounded values (user IDs, URLs with IDs), exploding storage and cost.
- Free-text logs with no request or trace ID.
- Logging passwords, tokens or personal data.
- Setting SLOs at 100%, or not at all.
Interview questions
Try answering out loud before you open each one.
How would you define reliability for this API?
Pick SLIs users care about: availability (share of non-5xx responses) and latency (share under a threshold, say 300 ms), measured at the load balancer. Set SLOs such as 99.9% over 30 days for each. Alert on fast burn of the error budget, and use the remaining budget to decide release pace.
Check yourself
Related topics
Further reading
- Service level objectives Google SRE book
- Signals: traces, metrics and logs OpenTelemetry documentation
Your notes
Saved in this browser as you type.
Finished this topic?
Mark it read to track your progress, then carry on with the next one.