Guide · Observability & Dynatrace

Observability for AI Workloads: What to Watch in Models, Agents and GPUs, and How Dynatrace Does It

An AI application fails differently from a web application. The answer is wrong rather than missing, the cost arrives per token rather than per server, the agent takes a path nobody wrote, and the GPU is the most expensive thing in the data centre. What to observe at each layer, why the old dashboards miss it, and how Dynatrace AI Observability covers models, agents, vector stores and GPUs in one platform with the cause worked out for you.

The short answer

Observability for AI workloads means seeing five things the classic stack does not: token usage and cost per request, model latency and errors across providers, the quality and drift of the answers, the path an agent took through its tools, and the GPUs underneath. Dynatrace AI Observability covers all five from one platform: OpenAI, Anthropic, Gemini, Amazon Bedrock, Azure AI Foundry and Vertex AI; agent frameworks including the OpenAI Agent SDK, LangChain and LangGraph, CrewAI, Bedrock AgentCore, MCP tools and Google ADK; the Milvus, Weaviate and Qdrant vector databases; and NVIDIA GPU telemetry, all ingested through OpenTelemetry and tied to the rest of your estate by Dynatrace Intelligence, which works out the cause rather than leaving you a dashboard. We are a Dynatrace partner, and this is the platform we would put in front of an AI estate.

Where we stand. C4C is an official Dynatrace partner. We advise on, license and implement Dynatrace, and this guide leans that way on purpose. The facts about what the platform does are drawn from Dynatrace's own documentation and announcements and are linked so you can check them.

Why AI workloads break the old observability model

A web service is up or down, fast or slow, and its cost is a server you already paid for. A model call is none of those things. It can return in two seconds with a confident wrong answer. It costs money on every request, in tokens, and a prompt change can double the bill without anyone touching infrastructure. An agent can call a tool, decide it needs another, call a model again, and take a route through your systems that no engineer designed, so a trace is the only record of what happened. Retrieval depends on a vector database whose relevance decays as the data changes. And the whole thing runs on GPUs that cost more per hour than anything else you own and sit idle more than anyone admits. None of that shows up as a red light on a dashboard built for request rate, error rate and CPU.

What to observe, layer by layer

LayerWhat to watchHow Dynatrace covers it
Models and providersToken consumption, latency, availability, errors and cost per request, per application and per team, across every provider you useAI Observability monitors token consumption, latency, availability and errors for OpenAI, Anthropic, Gemini, Amazon Bedrock, Azure AI Foundry, Vertex AI, NVIDIA NIM and Ollama, with cost tracked as token usage and service fees
Prompts and answersThe prompt flow from request to response, where latency sits, and whether the quality of the answers is driftingPrompt flow traced from initial request to final response, an audit trail of every input and output, and model and data drift tracked over time, with evaluations run through Arize AX using LLM as a judge and agent as a judge scoring
AgentsThe execution path, every tool call, the models called along the way, and agent to agent communicationVisibility into agent execution paths, tool invocations and inter agent communication for the OpenAI Agent SDK, LangChain and LangGraph, CrewAI, Amazon Bedrock AgentCore, MCP tools and Google ADK
RetrievalVector database performance and the semantic cache hit rate that decides both cost and relevanceMilvus, Weaviate and Qdrant, plus semantic caches, in the same traces as the model calls
GPUs and acceleratorsUtilisation, temperature and memory, and the share of expensive hardware that is doing nothingGPU and TPU metrics for NVIDIA GPUs, Google TPUs and Amazon inference hardware, including NVIDIA Data Center GPU Manager telemetry, mapped to the services running on them
Everything underneathThe Kubernetes cluster, the network, the database the agent just queried, and which of them caused the slow answerThe same Dynatrace platform that observes the rest of the estate, with Dynatrace Intelligence working out the cause from the live dependency map rather than correlating by time

Capabilities as documented by Dynatrace at the time of writing. Dynatrace for AI Observability, documentation. Supported providers, frameworks and integrations change often; check the current list before you design.

How Dynatrace AI Observability works

Open standards in, one platform out. Dynatrace ingests AI telemetry through OpenTelemetry with the GenAI semantic conventions and through OpenInference, the agent and model tracing specification that Arize donated to the OpenTelemetry project in 2026. Its earlier OpenLLMetry path reaches end of support on 31 January 2027, which is worth knowing if you instrumented with it. The practical result is that the application teams instrument once, with open libraries, and the data lands in the same platform as the infrastructure, the logs and the business events.

Agents and the GPUs they run on, in one view. On 2 July 2026 Dynatrace announced integration with the NVIDIA AI-Q Blueprint and the NVIDIA Agent Toolkit: the toolkit's OpenTelemetry traces are ingested and mapped to the infrastructure behind NVIDIA NIM and Nemotron microservices, so agent traces, token usage and GPU telemetry appear together. Dynatrace, observability meets NVIDIA AI-Q. That is the view a platform team needs when an agent is slow and the question is whether it is the model, the tool, the retrieval or the hardware.

The coding agents your developers run. Dynatrace also captures OpenTelemetry signals from coding agents such as Claude Code, Codex, Cursor and GitHub Copilot, so adoption, cost and reliability of the tools writing your software are visible in the same place as the software they write.

Cause, not correlation. The part that separates Dynatrace from a collection of dashboards is Dynatrace Intelligence, the causal AI that works out root cause from a live map of how every component depends on every other. For an AI workload that means the slow answer is traced back to the vector database node that is swapping, or the GPU that is thermally throttled, or the provider whose latency jumped, without a person stitching three tools together at two in the morning. Dynatrace from C4C.

The cost conversation

Token cost is the new cloud bill, and it arrives without a finance process around it. Observability that attributes tokens to applications, teams and features is how you find the prompt that doubled in length, the retry loop that triples every call, and the agent that calls the expensive model when the cheap one would do. Put that beside the GPU utilisation figure and you have the two numbers a board will ask about. Our AI inference cost calculator shows how fast the token line grows, and the Dynatrace pricing calculator shows what observing it costs at Dynatrace's published rates.

Why some teams reach for an open source stack, and why we would not

The AI teams building the first agents often start with Grafana, Prometheus and a tracing backend, because they are familiar, free to start and good at what they do. The question arrives at scale: three AI applications, two providers, a vector database, a GPU cluster and a production incident, and the answer is spread across four tools with a person as the correlation engine. Dynatrace's case is one platform, automatic instrumentation where it can, open standards where it should, and causal AI that finds the cause rather than showing you ten graphs. The honest comparison is in Dynatrace vs Grafana, and for an AI estate that is going to grow, the integrated platform is the one we would choose.

Where to start

  1. Instrument with open standards from the first application. OpenTelemetry GenAI conventions or OpenInference, not a vendor SDK, so the telemetry is portable and complete.
  2. Attribute tokens to owners on day one. Application, team and feature tags on every model call. Retrofitting cost attribution is painful.
  3. Trace the agents, not just the models. The tool calls and the path are where the surprises live.
  4. Put the GPUs on the same map. Utilisation per service, not per cluster, is the number that finds the idle hardware.
  5. Evaluate in production. Quality drift is a production signal. Score a sample of real traces continuously rather than trusting the pre release benchmark.
  6. Size the licence on the data you will keep. Traces from agents are verbose. Decide retention before the bill decides it for you. Dynatrace licensing, explained.

How C4C helps

We are a Dynatrace partner and an AI advisory practice, so we know both what the platform does and what an AI estate actually needs from it. We start with advice: which applications, agents and GPUs to bring under observation first, what the telemetry will cost to keep, and how it fits the observability you already run. Then we license, implement and tune Dynatrace for it, applying our IDEAL framework to identify the need, decide on the evidence, execute the rollout, make sure it is adopted and manage it across its lifecycle. Dynatrace implementation from C4C, and licensing sized to what you will use.

Running AI in production, or about to?

Tell us the applications, the providers and frameworks, whether agents are involved, and what you run on. Within two working days we will come back with what to observe first, what it would cost in Dynatrace at the published rates, and how it fits your existing tooling. Free, written, and no obligation.

Prefer email? Reach us directly at hello@c4cgroup.co.uk.

Frequently asked questions

What is AI observability?

Observability extended to the parts of an AI application the classic stack cannot see: token usage and cost per request, model latency and errors across providers, the quality and drift of answers, the path an agent takes through its tools, retrieval from vector databases, and the GPUs underneath. It uses the same traces, metrics and logs as the rest of observability, instrumented through OpenTelemetry, and it matters because an AI application fails by being wrong or expensive rather than by being down.

Does Dynatrace support AI observability?

Yes. Dynatrace AI Observability covers model providers including OpenAI, Anthropic, Gemini, Amazon Bedrock, Azure AI Foundry and Vertex AI, agent frameworks including the OpenAI Agent SDK, LangChain and LangGraph, CrewAI, Amazon Bedrock AgentCore, MCP tools and Google ADK, the Milvus, Weaviate and Qdrant vector databases, and GPU and TPU telemetry, ingested through OpenTelemetry and OpenInference. In July 2026 it added integration with the NVIDIA AI-Q Blueprint and Agent Toolkit so agent traces and GPU data appear together.

How do you monitor LLM token costs?

Instrument every model call with the OpenTelemetry GenAI conventions so each one carries its token counts, model and the application, team and feature that made it, then attribute cost in the observability platform. Dynatrace tracks token consumption and service fees per call and lets you break them down by owner, which is how you find the prompt that doubled in length or the retry loop that triples every request.

Can Dynatrace monitor AI agents and MCP tools?

Yes. It gives visibility into agent execution paths, tool invocations and inter agent communication for the main frameworks, including MCP tools, and traces the models called along the way. It also captures OpenTelemetry signals from coding agents such as Claude Code, Codex, Cursor and GitHub Copilot, so the adoption, cost and reliability of those tools are visible alongside everything else.

Do we need Dynatrace if we already use Grafana for AI observability?

Not on day one with one application. The case for Dynatrace arrives at scale, when several AI applications, providers, a vector database and a GPU cluster are in production and an incident has to be understood across all of them. One platform with automatic instrumentation, open standards and causal AI that finds the cause replaces the person stitching tools together. Our Dynatrace vs Grafana comparison sets out both sides.

What does AI observability cost?

In Dynatrace it is consumption based at published list rates, driven mainly by the volume of traces, logs and metrics you keep, and agent traces are verbose, so retention is the main lever. Our Dynatrace pricing calculator estimates it from hosts, data volumes and users at the published rate card, and we size the licence to what you will actually keep rather than the maximum.