Monitoring: What to Watch
You cannot fix what you cannot see. Learn what to monitor (health, availability, performance, security and usage), the four golden signals, the RED and USE methods, and why percentiles beat averages.
In a nutshell
- Monitor from the user's view first (is it up, is it fast, does it work), then the internals.
- The four golden signals: latency, traffic, errors and saturation.
- RED (rate, errors, duration) for services; USE (utilisation, saturation, errors) for resources.
- Use percentiles (p50, p95, p99), not averages: the average hides the users having a bad time.
Five things to monitor
| Area | Question | Examples |
|---|---|---|
| Health | Is each component working? | Health checks, crash loops, dependency status |
| Availability | Can users use it? | Successful requests ÷ total, external uptime checks |
| Performance | Is it fast enough? | Latency percentiles, throughput, queue age |
| Security | Is anyone attacking or abusing it? | Failed logins, unusual traffic, permission changes |
| Usage | How is it used, and what will we need? | Active users, feature use, storage growth, cost |
The four golden signals
Google's Site Reliability Engineering book suggests that if you can only measure four things about a user-facing system, measure these:
- Latency: how long requests take. Track successful and failed requests separately; a fast error is not a fast success.
- Traffic: how much demand there is (requests per second, messages per second).
- Errors: the rate of failed requests, including "successful" responses with wrong content.
- Saturation: how full the system is: CPU, memory, connection pools, queue length. Saturation predicts trouble before latency shows it.
Two related checklists: RED (Rate, Errors, Duration) for every service endpoint, and Brendan Gregg's USE method (Utilisation, Saturation, Errors) for every resource such as CPU, disks and network.
Percentiles, not averages
If 99 requests take 50 ms and one takes 5 seconds, the average is about 100 ms: it looks fine and describes nobody. Percentiles describe the distribution: p50 (median) is the typical request; p99 is the slowest 1%.
- p50half of requests are faster than this
- p951 in 20 requests is slower
- p991 in 100 requests is slower
Black box and white box
Watching from outside and inside
Diagram as text
- Uptime probe → Load balancer: GET /
- Load balancer → Services
- Services → Database
- Services → Monitoring: white box
- Database → Monitoring
Black-box monitoring tests the system from outside, like a user: it catches DNS, certificate and network problems that internal metrics never see. White-box monitoring uses data the system reports about itself, to explain causes and spot problems before users notice.
Common mistakes
- Alerting on CPU and memory but never on user-facing errors or latency.
- Watching averages instead of percentiles.
- No external check, so an expired TLS certificate goes unnoticed.
- Collecting thousands of metrics nobody looks at.
Interview questions
Try answering out loud before you open each one.
What would you monitor for this service?
Start with user-facing signals: request rate, error rate and latency percentiles per endpoint, plus external uptime checks. Then saturation: CPU, memory, connection pools, queue length and age, and the health of dependencies (database latency, cache hit ratio). Add business metrics (orders per minute) that catch silent failures.
Check yourself
Related topics
Further reading
- Monitoring distributed systems (the four golden signals) Google SRE book
- The USE Method Brendan Gregg
Your notes
Saved in this browser as you type.
Finished this topic?
Mark it read to track your progress, then carry on with the next one.