A few years ago, I watched a team spend nearly an hour proving that their system was healthy.

CPU usage looked normal. Memory was fine. The database was responding. Every dashboard was green.

Meanwhile, customers couldn’t complete payments.

The problem eventually turned out to be a small change in the way one service passed an order reference to another. Nothing had crashed. No server was on fire. The application was doing exactly what the monitoring system expected while quietly failing at the thing customers actually needed.

That incident changed how I think about system health.

Monitoring is useful. We absolutely need it. But a wall of healthy charts can create a very convincing lie.

The alert was correct and still not helpful

Most monitoring is built around questions we already know how to ask.

Is the server running?

Has the error rate crossed five percent?

Is the database using too much memory?

Did the response time exceed two seconds?

These questions are sensible. When something predictable goes wrong, monitoring catches it quickly. A disk becomes full, an API stops responding or traffic suddenly jumps. An alert appears in Slack, someone opens a dashboard, and the investigation begins.

The problem is that modern software rarely fails in such a polite way.

A food delivery app can accept payment, show the order as confirmed and still fail to send it to the restaurant. A bank transfer may succeed in one system but remain “processing” in the customer’s app. A user might log in normally through Gmail but get stuck when using a LinkedIn account.

Each component can look healthy on its own.

This is why observability has become more valuable. Monitoring watches known conditions. Observability gives engineers enough evidence to investigate behaviour they didn’t predict. AWS describes monitoring as measuring system health, while observability examines interactions across a distributed system to find the cause of unusual behaviour.

That sounds like a small difference. During an incident, it’s enormous.

Software no longer lives in one place

Older business applications were often easier to picture. One application, a database and perhaps a separate server handling email.

Now consider what happens when someone clicks “Place order” on a shopping app.

The request may touch authentication, pricing, inventory, fraud detection, payments, delivery estimates, notifications and a third-party courier service. Some parts run in different cloud regions. A few are managed by outside companies. One still depends on a database nobody wants to replace because it contains eight years of customer history.

This setup works surprisingly well most days.

Then something slows down.

Monitoring might report that checkout is taking six seconds. Useful. But which six seconds? Did the payment provider respond late? Was the inventory service waiting for a lock? Did a new release send three requests where it used to send one?

Without context, engineers start opening dashboards and guessing.

I’ve seen this turn into a strange group ritual. Five people join a call. Someone shares a screen. Everyone asks whether anything changed recently, even though software teams change things constantly. Then one person searches logs while another checks deployments and somebody else quietly looks through old Slack threads.

Observability doesn’t eliminate this mess, but it gives the team a better trail.

The useful part is the connection

People often describe observability through metrics, logs and traces.

Metrics show patterns: response times, failed requests, memory use, queue length and so on.

Logs record events and details from individual components.

Traces follow one request as it travels through several services. OpenTelemetry describes a distributed trace as the path taken by a single user or application request across a multi-service system.

None of these is especially magical alone.

The value comes from connecting them.

Suppose a customer reports that an online banking transfer failed at 10:42. You find the request, follow its trace and notice that the identity service responded normally. The transfer service then waited four seconds for a fraud check. A related log shows that the fraud provider rejected a field introduced in that morning’s release.

Now you have a story.

Without that connection, you may have thousands of log lines, twelve graphs and an alert saying transfers are slow. Technically, plenty of data. Practically, not much understanding.

Here’s the thing: collecting information is not the same as being able to investigate with it.

More data can make you less informed

Observability platforms are sometimes sold as if the answer is to collect everything.

Every log. Every request. Every event. Forever.

That approach becomes expensive very quickly, and it can make incidents harder to investigate. Searching through millions of nearly identical messages is not visibility. It’s digital fog.

Teams should collect data that helps answer real operational questions. Which customer was affected? Which version handled the request? What dependency was involved? Did the failure begin after a deployment? Can we follow the same transaction across services?

The system also needs consistent identifiers, useful timestamps and sensible context. OpenTelemetry exists partly to standardise how applications generate and export traces, metrics and logs across different tools and platforms.

If your logs say only “Request failed,” no expensive dashboard can rescue them.

The funny part is that teams often buy a new observability platform before fixing the messages their applications produce. It’s like purchasing better search software for a folder full of files named final_v2_revised_new.

Alerts should point to people, not machines

A server being busy is not automatically a problem.

If customers are completing purchases normally, a brief CPU spike may not deserve a 2 a.m. notification. Yet many teams still alert on infrastructure numbers because they are easy to measure.

The better question is whether users are experiencing harm.

Can they log in? Are payments completing? Are messages being delivered? Is the service meeting the response time people actually notice?

Monitoring should remain part of this. I don’t agree with the fashionable idea that observability somehow replaces it. You still need alerts for known, urgent failures. Nobody wants an investigative journey to discover that the production database is offline.

But alerts should open the door to an investigation rather than simply announce that a number moved.

A useful alert includes context: which service changed, which customers are affected, where the request slowed down and whether a recent deployment is connected. It helps the person receiving it decide what to do next.

Otherwise it is just noise with a timestamp.

This is really about shortening confusion

Companies talk about observability in terms of uptime, incident response and reliability. All true.

But its everyday value is simpler: it reduces the amount of time engineers spend being confused.

It turns “checkout is weird” into “requests from the mobile app are timing out while waiting for the inventory service.” It gives support teams something more useful than “engineering is investigating.” It helps developers see how code behaves after leaving the clean safety of a laptop.

Monitoring tells you that the system has entered a state you expected enough to measure.

Observability helps when reality invents a new one.

Most production failures live somewhere between those two sentences. That’s why the green dashboard isn’t enough anymore.