AgentOps Grew Up: OpenTelemetry Won the Agent Observability War

Ask an engineer to debug a failed agent run in 2025 and you would watch them scroll. Through a JSON blob of messages, through provider dashboards that each showed one slice, through logs that recorded the request and not the reasoning, trying to reconstruct why a nine-step task went wrong at step four. The tooling existed and none of it agreed on what a step was. That changed, and the thing that changed it was not a new product. It was a schema.

Why Agents Broke Conventional Observability

Traditional distributed tracing assumes a request fans out through services and comes back. The shape is wide and shallow, the work at each hop is deterministic, and a failure has a location. Agent execution has none of those properties. It is deep rather than wide, it loops, the number of steps is not known in advance, and the interesting failures are semantic rather than structural.

Consider what "the agent did the wrong thing" means. Every HTTP call returned 200. Every tool executed successfully. No exception was thrown, no latency budget was blown, no error rate moved. The agent simply chose to call the search tool with a query that was subtly wrong, got plausible results, and built nine subsequent steps on top of them. Conventional observability is blind to this by construction, because it instruments whether things happened rather than what they contained.

What the Conventions Actually Standardise

The GenAI semantic conventions cover four areas, and the split is worth internalising because it maps cleanly onto the questions you will want to ask later.

LLM client spans wrap a single model call: the model name, the provider, token counts in and out, temperature and other parameters, finish reason, and latency. This is the unit that costs money, and standardising it is what makes cost attribution possible across providers. Agent spans wrap a reasoning unit: an agent invocation with a name, a task, and the child spans it produced. This is the unit that answers "what was it trying to do."

Events carry prompt and completion content attached to spans rather than stuffed into attributes, which matters for both size limits and for redaction policy. Metrics aggregate the whole thing: token counts, call counts, latencies, and error rates, sliced by model and operation. Tool executions and MCP calls get their own span treatment, so a trace shows not only that a tool ran but which server served it and how long it took.

That last development deserves attention beyond its convenience. When the coding agents your engineers use every day emit standard telemetry, agent activity stops being a separate observability domain and becomes part of the same trace graph as everything else. A slow build, a flaky test, and an agent that burned 400,000 tokens exploring the wrong directory all show up in one place, queryable with one language.

Cost Attribution Is the Feature That Sells It

Debugging is the reason engineers want this. Cost attribution is the reason it gets funded, and it is genuinely the larger prize. Most organisations running agents at any scale can tell you their total monthly model spend and almost nothing else. Which feature consumed it, which team, which customer, which specific workflow doubled last month: all unanswerable, because the only available dimension is the API key.

With token counts on standardised spans, and with those spans carrying your own resource attributes for team, feature, and environment, cost becomes a normal dimension of a normal query. You can rank workflows by spend, find the retry loop that accounts for a fifth of your bill, and tell a product owner what their feature costs per active user. That last capability changes conversations. A feature that costs eleven cents per use is a different product decision than one that costs three tenths of a cent, and without attribution nobody in the room knows which one they are discussing.

This directly addresses the dynamic we covered in token pricing breaking enterprise AI budgets, where organisations burned annual budgets in a third of the year. The failure there was rarely overspending on purpose. It was spending without a per-unit view, discovering the total at month end, and having no way to identify what to change. Trace level cost data converts that from a finance surprise into an engineering metric with an owner.

What to Instrument First

Full trajectory capture on every run is expensive in storage and in review attention. Start narrow. Four things earn their keep immediately.

Token counts on every model call, always

Non-negotiable and cheap. Input tokens, output tokens, cached tokens where the provider reports them, model name, and your own team and feature attributes. This alone delivers the cost attribution above and takes an afternoon.

Tool calls with arguments and outcome

Tool selection is where agents most commonly go wrong, and the argument values are the diagnostic. Capture which tool, what arguments, whether it succeeded, and how long it took. Redact argument values by field rather than wholesale, since a tool call without arguments tells you almost nothing about why the run went sideways.

Step count and termination reason per run

How many steps did this run take, and why did it stop: success, step limit, token limit, error, or human intervention. Step count distributions are the single best early warning for agent degradation, because a model or prompt change that makes the agent less efficient shows up here days before it shows up in quality complaints or in the invoice.

Full content capture, sampled and on failure

Prompts and completions are the expensive signal in both storage and sensitivity. Capture them on every failed run, on a small random sample of successes for baseline comparison, and on anything a user flagged. That combination gives you the material to debug without retaining conversation content at a volume your security review will object to.

What a Useful Agent Trace Looks Like

Descriptions of tracing tend to stay abstract, so it helps to walk one failure through. A support agent is asked to find a customer's most recent invoice and explain a charge. The run takes nineteen steps and ends with a confident answer about the wrong invoice. Nothing errored.

In the trace, the top-level agent span shows nineteen children and a termination reason of success. Step three is a tool span for a customer lookup, and its arguments show the agent searched by display name rather than by customer ID, because the ID was never in its context. Step four returns two matching customers. Step five is an LLM client span whose completion picks the first one, with no indication in the reasoning that a choice was made at all. Steps six through nineteen are all correct work performed against the wrong customer.

Every diagnostic fact there came from a different signal. The step count told you it was a long run. The tool arguments told you what was searched. The tool result told you the query was ambiguous. The completion content told you the ambiguity was never surfaced. Remove any one of the four and the investigation stalls, which is the argument for instrumenting all four rather than starting with whichever is easiest.

The fix that follows is also visible from the trace: the customer lookup tool should return an explicit ambiguity result rather than a list, forcing the agent to disambiguate rather than choose. That is a tool design change, not a prompt change, and teams without traces almost never find it. They add a line to the system prompt asking the model to be careful, which works sometimes and is impossible to verify.

Instrumenting Against a Moving Specification

The conventions are formally in Development, and the GenAI and MCP portions were still there as of May 2026. Attribute names have changed before and will change again. The correct response is not to wait for stability, because the specification stabilises by being used, and because the alternative to an imperfect standard is a proprietary format that will definitely need migrating.

Write your instrumentation behind a thin internal module that maps your domain concepts onto convention attribute names in one place. When a name changes, you edit one file rather than grepping a codebase. This is a twenty-line abstraction that has saved teams a week, and it is the same reasoning that makes a routing layer worth building around models.

Prefer emitting through the OTLP protocol to a collector you control rather than directly to a vendor. The collector gives you a place to redact, sample, and fan out to more than one backend, and it means changing observability vendors is a configuration change. Given how quickly this tooling market is consolidating, that optionality is worth the extra component.

Sampling and Retention Without Going Broke

The bill arrives faster here than teams expect, and the reason is a ratio nobody checks in advance. A conventional service emits perhaps fifteen spans per request. An agent run emits sixty, some of them carrying multi-kilobyte prompt and completion content. Instrument agents the way you instrument services and your observability spend can rival your inference spend, which is an embarrassing outcome for a project justified on cost visibility.

The policy that works separates the cheap signals from the expensive ones and treats them differently. Metrics, token counts, step counts, latencies, and outcomes are small, aggregate well, and should be kept at full fidelity for a long window. They are what you query for trends and what powers cost attribution, and they compress to almost nothing.

Span structure without content is the middle tier: keep every span for a shorter window, because trace shape is what tells you an agent looped nineteen times, and you cannot reconstruct that from metrics. Content is the expensive tier and belongs on a sampling policy: everything on failures, everything on user-flagged runs, and a small percentage of successes as a baseline. Ten percent is generous for most teams and one percent is workable at high volume.

Two refinements are worth the effort. Sample by trace rather than by span, since half a captured agent run is worth much less than none of one and costs half as much as the whole. And make the sampling decision at the head of the trace with a deterministic function of the trace ID, so every service in the path makes the same choice and you do not end up with fragments.

Retention deserves an explicit decision rather than a default. Content carrying customer data should expire faster than your general observability retention, both for cost and because a shorter window is much easier to defend in a privacy review. Set that at the collector and it applies uniformly, rather than depending on each backend's configuration being remembered correctly.

Build, Buy, or Both

The standardisation changed this decision more than teams have noticed. Before the conventions, choosing an LLM observability vendor meant adopting their SDK, their span model, and their mental model, and switching later meant reinstrumenting. That is what made the choice feel weighty. With standard emission through a collector you own, the backend becomes a consumer of data you already produce.

The practical consequence is that you should emit standard telemetry regardless, and then choose a backend on ordinary criteria: query ergonomics, retention cost, whether your existing on-call already lives there, and whether the AI-specific views are genuinely better than what you would build on a general tracing tool. For most teams already running a mature observability stack, the answer is to send agent traces to the same place as everything else and add a dedicated tool only if a specific need appears.

The specialist tools earn their place in two situations. If your core product is agentic and prompt iteration is a daily activity across a non-engineering team, the prompt-centric interfaces are worth real money. And if you need trajectory-level eval scoring wired directly to production traces, closing the loop between what shipped and what was measured, that integration is meaningful work you would otherwise build. Outside those cases, a second backend is mostly a second bill.

The Questions You Can Finally Answer

Instrumentation is worth judging by the questions it makes answerable, so here is the concrete list. Which workflow consumed the most tokens last week. Has the average step count for our highest-volume task changed since the model update. Which tool has the highest failure rate, and does that failure correlate with runs that eventually failed overall. What does this feature cost per active user. Which customer's usage pattern is responsible for the tail latency our support team keeps hearing about. Are agents retrying the same failing call in a loop anywhere.

Every one of those is routine with standardised traces and close to impossible without them. And every one of them is a question somebody in your organisation has already asked and received a shrug in response to.

There is a broader point here about where agent reliability actually comes from. The pattern we described in agents that never reach production reliability is that the model is rarely the binding constraint. What separates a demo from a system is the surrounding infrastructure, and observability is the part of that infrastructure that makes every other part improvable. You cannot fix an escalation rate you cannot see, tune a step budget you do not measure, or justify an eval investment without evidence of what is failing. Tracing is the layer that turns the rest of the agent stack from a set of opinions into a set of measurements.