A system can be "green" on every dashboard and, at the same moment, keep one user in a hundred waiting ten seconds. Classic monitoring answers questions you know in advance: is the server up? Is the disk full? Observability is for the questions you hadn't foreseen: why are those particular requests slow, and only since yesterday?
Monitoring and observability
Monitoring checks known indicators and raises alerts when they cross a threshold. It is indispensable, but it assumes you already know what can go wrong.
Observability is a property of the system: how well you can understand what is happening inside it by looking at the data it produces. An observable application emits information rich enough to let you investigate new problems without having to add code and ship a new version.
The data it relies on is traditionally of three kinds, often called the three pillars.
Metrics
Metrics are numerical values measured over time: requests per second, error percentage, response time, memory used, queue length. They are compact, cheap to keep for a long time and ideal for charts and alerts.
For response times the average is misleading. If ninety-nine requests take 50 milliseconds and one takes ten seconds, the average looks acceptable but one user in a hundred has a terrible experience. That is why percentiles are used: the 95th or 99th percentile indicates the time within which 95 or 99 percent of requests complete.
A practical criterion for choosing what to measure on a service is to watch four signals: traffic, errors, latency and resource saturation.
The limit of metrics is cardinality. They can be broken down by labels, such as endpoint or response code, but adding labels with too many distinct values, such as the user ID, makes the number of series to store, and the costs, explode.
Logs
Logs are records of events: a request received, an error, an operation completed. They contain the detail that metrics lack: which user, which parameter, which error message.
To be useful in a distributed system, logs must be:
- structured, that is in JSON or a similar format, with separate fields instead of free sentences, so you can filter by field;
- centralized, collected from all servers into a single queryable place;
- correlatable, with a request identifier that lets you follow the same operation across different components;
- restrained, with no passwords, tokens or unnecessary personal data.
A structured log looks like this:
{"ts":"2026-03-14T10:22:31Z","level":"error","service":"orders","request_id":"a91f3c","msg":"timeout calling payments","duration_ms":5003}
Distributed traces
In an architecture with many services, a single user request can pass through five or six of them. If it is slow, which service is responsible? Traces answer this question.
A trace follows a request from start to finish. It is made of spans, each representing an operation: a call to another service, a database query, a cache read. Each span has a start, a duration and a reference to the span that generated it. Visualized, a trace shows at a glance where the time goes: for example, that a database query takes four of the five seconds in total.
Traces are expensive to keep in full, so sampling is used: only a portion of requests is recorded, choosing to always keep those with errors or that are particularly slow.
Using them together
The strength of observability lies in moving from one type of data to another.
- A metric reports that the 99th percentile of latency has risen.
- Filtering the traces from that period reveals that the slow requests all go through the same service.
- That service's logs, filtered by the trace identifier, show the exact error message.
For this to work, logs and traces must share the same identifier, and all data must use the same names for services and environments.
OpenTelemetry
For years every observability vendor had its own libraries, tying applications to one product. OpenTelemetry is an open source project that defines a common standard for generating and transmitting metrics, logs and traces. You instrument the code once and can send the data to different systems, open source or commercial, by changing only the configuration.
Many languages also offer automatic instrumentation, which records HTTP calls and queries to the most common databases without code changes.
Where to start
You don't need complex infrastructure to begin. A reasonable path:
- expose the service's essential metrics and create a few charts on the four signals;
- switch to structured logs and add a request identifier;
- collect the logs in a single place;
- introduce traces when there are more than two or three services.
In short
- Monitoring checks known problems; observability lets you investigate unexpected ones.
- Metrics are numbers over time: use percentiles for latency and watch out for cardinality.
- Logs must be structured, centralized, correlatable and free of sensitive data.
- Traces show a request's path across services and where time is lost.
- OpenTelemetry offers a single standard for producing the data without vendor lock-in.
