Why most enterprises instrument the wrong tier

By Nick Watson, Chief Technologist

Ask most large organisations how they decided what to instrument and how deeply, and the honest answer is that they did not decide. The coverage they have is the coverage they ended up with, and it is almost never the coverage they would choose.

There are two ways to get it wrong, and most estates manage both at once.

The everything at full depth trap

The first is instrumenting everything at the deepest setting because the platform allowed it. An agent goes onto every host, tracing is turned on for every process, and the estate begins producing an enormous volume of telemetry. Some of it is useful. Most of it is data about machines nobody worries about, kept at a fidelity nobody needs, at a cost nobody planned for. The dashboards fill with services that have never caused an incident, and the signal that matters gets harder to find, not easier.

The instinct behind it is understandable. Full depth feels safer than gaps. But depth everywhere is not the same as visibility where it counts, and it usually buys the opposite.

The project servers trap

The second is the reverse. Instrumentation went onto the handful of servers a particular project cared about, because that project paid for the work and scoped it to its own estate. Everything outside that scope was left as it was. Then an incident starts on a service that no project ever adopted, and there is nothing to look at. The team that owns the outage discovers, in the middle of it, that the one system they need to see was never on anyone's list.

Both traps have the same root. The coverage decision was made by whoever installed the agent, or by whichever budget paid for the rollout, rather than by anyone who understood the application and what it is worth.

Start from the application, not the server list

The fix is to make the decision on purpose, and to start it from the right place. Not the inventory of servers, which tells you what exists but not what matters, but the application and the business it serves. Which services carry money or risk. Which are supporting cast that need to be healthy but rarely need deep inspection. Which are inventory that simply needs to be known to exist.

From there, every host, container and service is placed in one of three tiers. Full depth, for the services that carry value or risk, where a slow request has a consequence worth following inside the code. Infrastructure only, for the supporting cast, where the health of the machine is enough. And discovered, for everything that needs to be on the map but no more than that.

The point of the exercise is not the diagram. It is that someone can now explain why each part of the estate is where it is, and change it deliberately when the business changes. Coverage stops being an accident and becomes a decision you can defend.

Three questions before you decide a tier

For any server, container or service, three questions settle its tier faster than any inventory:

• If this failed right now, what would it cost, and who would feel it first?
• If it were merely slow rather than down, would anyone notice before a customer did?
• When it does go wrong, will we need to see inside the process to fix it, or is knowing the machine is unhealthy enough?

A service that costs real money when it slows, that customers feel before you do, and that needs inside level detail to diagnose, belongs at full depth. A service that fails none of those tests does not, and instrumenting it as though it does is how estates end up expensive and noisy at the same time.

None of this is a tooling problem. Modern platforms will happily instrument at any tier you ask for. The failure is in never being asked. Deciding the tiers on purpose, from the application down, is the single highest return move in any observability programme, and it is the one most often skipped.

Read more on instrumentation design →