Observability & Metrics
The problem
In a system with many moving, independently-failing parts, you can't just look at it and see what's wrong — observability is the discipline of instrumenting a system so its internal state is inferable from the outside, through logs, metrics, and traces, before and during an incident, not just after.
Why now
Logging and system monitoring already covered logs and point-in-time resource snapshots on a single machine; this topic extends that to a distributed system's health as a whole, and probability's percentile-based thinking is exactly the vocabulary needed to describe latency (p50, p99) meaningfully instead of misleadingly with a single average.
Mental model
Metrics answer 'what's happening in aggregate' (request rate, error rate, latency percentiles), logs answer 'what happened in this specific case,' and traces answer 'where did the time go across this one request's journey through multiple services' — the three together are what make an unfamiliar failure diagnosable instead of a total mystery, and p99 latency specifically answers 'how bad is the worst 1% experience,' which an average completely hides.
Requires
- Logging & System MonitoringLinux
Observability generalizes the single-machine logging and monitoring practices already covered to a distributed system with many services, adding metrics aggregation and distributed tracing on top of that same foundation.
- Probability BasicsMathematical Thinking
Latency percentiles (p50, p99) are a direct, practical application of the probability distribution thinking already covered — a p99 figure is a precise statement about the shape of a distribution, not just a bigger average.
Unlocks
Used in
- Prometheus, Grafana, and Datadog as standard observability tooling
- On-call incident response, relying entirely on metrics, logs, and traces to diagnose an active outage
- SLOs (Service Level Objectives) defined directly in terms of latency percentiles and error rates
- Distributed tracing (Jaeger, OpenTelemetry) following one request across many microservices
Projects
- Instrument a small service with basic metrics (request count, error count, latency) and build a dashboard visualizing them over time
- Explain, using a real latency distribution, why the p99 latency can be 10x the average (p50) and what that implies about the worst-case user experience
Examples
- A service with a 50ms average latency but a 2-second p99 means 1% of users have a genuinely bad experience the average completely hides
- A distributed trace showing one request spending 800ms of its 1-second total in a single downstream service pinpoints exactly where to optimize
Resources
Mastery checklist
- I can explain the distinct roles of logs, metrics, and traces in observability
- I can instrument a service with basic request/error/latency metrics
- I can explain why p99 latency is more informative than average latency for user experience
- I can use a trace to identify which part of a multi-service request is slow