A team I worked with once had twelve monitoring dashboards open during an outage.

CPU graphs. Database latency. Application errors. Network traffic. Cloud metrics. Customer experience. Queue depth. Another dashboard whose owner nobody could identify.

The room looked very informed.

Nobody knew what was wrong.

That was the moment I stopped believing that more observability automatically creates more understanding.

Modern systems generate enormous amounts of telemetry. Metrics, logs, traces, events and alerts can reveal what software is doing in remarkable detail.

They can also bury the useful signal under thousands of technically correct facts.

Observability is valuable.

Observability theatre is not.

Collecting everything feels responsible

When storage is cheap and monitoring tools make collection easy, the default becomes “send it all.”

Every service exports metrics. Every request produces logs. Every deployment generates events. Distributed tracing follows transactions across dozens of components.

This sounds sensible because nobody wants to discover, during an incident, that the one missing field was exactly what they needed.

So teams collect more.

Then the incident arrives and the engineer searches through enough data to describe a small planet.

Information only helps when people can move from symptom to cause.

A million searchable log lines are not automatically better than twenty meaningful ones.

Alerts teach people to ignore alerts

The first week after installing a monitoring system, every alert feels important.

Three months later, the team has learned that the warning at 2:14 a.m. usually clears itself.

Someone mutes a noisy channel. Another person creates an email filter. The on-call engineer sees five notifications before breakfast and assumes the sixth is similar.

Then the real incident arrives.

Alert fatigue is not a failure of employee discipline. It is a system-design problem.

If an alert does not require action, why is it interrupting a person?

Warnings can still be useful on dashboards or reports. Pager-level alerts should mean somebody needs to make a decision now.

Humans are extremely trainable.

If you teach them an alarm is meaningless, they will learn.

The customer symptom should be visible

Infrastructure teams naturally monitor infrastructure.

CPU. Memory. Disk. Containers. Network.

Customers care whether checkout works.

These views can diverge.

A service may report healthy CPU usage while every customer request fails because an external authentication provider is down. A database can look normal while a bad deployment sends the wrong data to users.

I like monitoring that begins with outcomes.

Can users log in? Can they complete a purchase? Are important API requests succeeding? Is latency within the range people actually feel?

Then drill into technical metrics.

The infrastructure exists to support a service.

The dashboard should remember that.

Logs need sentences humans can use

Developers often write logs while thinking about the code in front of them.

`Error processing request`.

Wonderful.

Which request? Which customer? Which dependency failed? Was the operation retried? What happened immediately before?

Good logs provide context without dumping sensitive information.

Use structured fields. Include request identifiers. Record meaningful state transitions. Avoid logging passwords, tokens or personal data because debugging should not become a privacy breach.

Also, log levels matter.

If everything is an error, nothing is.

A healthy application should not produce a red waterfall during normal operation.

Tracing becomes useful when systems become distributed

In a simple application, one request may enter one server and reach one database.

In modern architectures, a request can cross several services, queues and third-party APIs.

A customer clicks Pay.

The frontend calls an API. The API calls an order service. The order service talks to inventory and payment systems. A background job sends confirmation.

Where did the delay happen?

Distributed tracing can follow one transaction across these boundaries and show how long each step took.

This is incredibly helpful when the architecture deserves it.

Adding tracing to a tiny application with three endpoints may create more instrumentation than insight.

Tooling should match complexity.

Not every bicycle needs air-traffic control.

Dashboards should have an audience

A dashboard built for a database engineer will not necessarily help a product manager.

That sounds obvious. Organisations still create giant shared dashboards containing everything because “single pane of glass” sounds efficient.

People need different questions answered.

An on-call engineer wants current health and recent changes. A service owner wants trends. Leadership may care about reliability, customer impact and whether recurring problems are improving.

Design dashboards around decisions.

If nobody can name the decision a chart supports, remove it.

The chart can come back later if somebody misses it.

This is surprisingly rare.

Dashboards accumulate like browser tabs.

AI can summarise incidents, but it needs evidence

AI tools are being added to observability platforms to summarise alerts, search logs and suggest likely causes.

This can help enormously.

During an incident, nobody wants to manually correlate twenty signals if a system can point out that errors began immediately after a deployment and are concentrated in one service.

The danger is confidence.

An AI summary should link to the metrics, logs and traces supporting the conclusion. Engineers need to inspect the evidence, especially before rolling back systems or changing production configuration.

A fluent explanation is useful.

A fluent explanation with no traceable evidence is another alert.

I want assistants that reduce search, not assistants that replace verification.

Incidents need timelines more than screenshots

During a serious outage, teams often take screenshots of graphs and paste them into chat.

Useful in the moment.

Later, the post-incident review becomes archaeology.

A better practice is to build a timeline.

When did the first customer impact appear? What changed beforehand? Which alerts fired? What actions did the team take? When did recovery begin?

This turns telemetry into a story.

The goal is not to find someone to blame.

It is to understand how the system behaved and whether monitoring helped people notice quickly.

Observability should improve after every incident.

Otherwise the dashboards are merely recording history.

Delete metrics occasionally

This feels wrong because engineers like keeping data.

But monitoring systems need gardening.

Remove obsolete alerts. Archive dashboards for retired services. Stop collecting high-volume logs nobody uses. Review retention periods. Rename confusing metrics.

Every piece of telemetry has a cost.

Storage money is one part. Query performance is another. Human attention is the expensive part.

A smaller monitoring system with trusted signals can outperform a giant one nobody understands.

Reduction is a legitimate observability project.

The useful question is still simple

When an incident begins, people want to know three things.

What is broken?

Who is affected?

What changed?

A good observability setup helps answer those quickly and then lets engineers dig deeper.

The twelve dashboards from that outage were eventually consolidated. Several alerts were deleted. Customer-facing checks became more prominent. Deployment information was placed beside service health.

The team collected slightly less data.

They understood the next incident faster.

That is the metric I care about.