Observability Tooling for Agents

Product choice, not architecture

Building the actual tracing architecture -- spans per LLM call and per tool call, parent-child relationships across a plan-execute-reflect loop -- is covered by this series' observability and tracing architecture article. This one is narrower: given that architecture, which product should capture and store the resulting traces? The honest answer changes depending on whether a team already has observability infrastructure worth extending, or is starting from nothing.

Purpose-built LLM observability platforms

Products in this category (LangSmith and Langfuse are the reference examples; Arize represents the ML-observability-first end of the same space) are built around the specific shape of LLM traffic: a trace is a tree of prompt/response pairs, each carrying token counts, cost, latency, and often the full message history that produced a given completion. They ship with UIs purpose-built for reading that shape -- diffing two runs of the same prompt, filtering traces by cost or latency outliers, and replaying a specific failing trace with a modified prompt to see if a fix would have worked.

That last capability -- prompt-level replay against a captured trace -- is the feature that's genuinely hard to replicate with generic infrastructure, because it requires the tool to understand that a span is specifically an LLM call with a prompt template and variables, not just an opaque unit of work. If prompt iteration against real production failures is a frequent activity for the team, this category earns its keep specifically for that workflow.

The cost is a new vendor dependency and, for the hosted versions of these products, another system with access to potentially sensitive prompt/response content -- the same data-locality consideration that applies to any managed service handling application data.

Building on OpenTelemetry

The alternative is treating an LLM call as just another span in a standard OpenTelemetry trace, using the same collector, backend, and dashboards already handling the rest of the application's observability. Concretely: wrap each LLM call and tool call in a span, attach prompt, response, token count, and cost as span attributes, and let the existing trace-visualization tooling show the resulting parent-child tree alongside spans for database calls, HTTP requests, and everything else already instrumented.

What this buys: one observability stack instead of two, and traces that show an LLM call in the same timeline as the database query or downstream API call that surrounds it -- useful when the actual latency culprit in a slow agent run turns out to be a tool call, not the model, and a unified trace makes that visible at a glance instead of requiring a correlation across two separate systems.

What it costs: the prompt-specific UX -- diffing prompt variants, replaying a trace with a modified prompt -- has to be built by hand if it's wanted at all, and most generic tracing backends have no native concept of "this span's cost was $0.003" without custom dashboarding on top of the raw attribute.

What to actually capture, regardless of which product you pick

The tool matters less than the discipline of capturing the same fields consistently, because a trace missing one of these fields is a trace that can't answer the question it was built for.

per LLM-call span:
  prompt (template id + resolved variables, not just final string)
  response (raw, before any post-processing)
  model + provider, token counts (input/output), cost, latency
  parent span id (which plan step / tool call triggered this)

per tool-call span:
  tool name, arguments, result, success/failure, latency
  retry count if a retry policy fired

per run (root span):
  goal / user request, final outcome, total cost, total latency,
  replan count if applicable

The prompt template id plus resolved variables (not just the final rendered string) is the detail most setups skip and most regret skipping -- it's what makes "show me every trace that used prompt version 14" a query instead of a manual grep, and it's the exact field a purpose-built platform's replay feature depends on. Capture it from day one, on either kind of tooling, and the choice between the two stops being a decision you can't walk back.

The actual decision point

Teams sometimes run both, deliberately: OpenTelemetry for the system-wide trace that shows an agent run in context with everything around it, and a purpose-built platform layered on top specifically for the prompt-engineering workflow the generic tooling doesn't serve well. That's not redundancy if the two are actually used for different tasks -- it's redundant only when nobody has decided which tool answers which question, and both end up half-used.

Debugging a wrong answer by walking the trace

Concretely: a support agent is asked "can I get a refund on order 4471," and it answers "no, that order isn't eligible" -- wrong; the order was eligible. Without a trace, the only options are re-running the same request and hoping the failure reproduces, or reading application logs that weren't written with this specific question in mind. With a trace tree, the investigation is a walk down the span hierarchy.

The root span shows the goal and the final answer. Its children show the plan: a lookup-order tool call, a check-refund-policy tool call, a final generation step. The lookup-order span shows the tool was called with order_id=4471 and returned the order's actual data -- so the data fetch wasn't the problem. The check-refund-policy span shows the tool was called with the order's purchase date as an argument, but the policy tool's contract expects the delivery date for the eligibility window -- a wrong argument, visible directly in the span's captured arguments, not something that would show up in a log line reading "checked refund policy." The generation span downstream shows the model correctly reasoning from the (wrong) policy-check result to a (wrong but locally consistent) answer -- the model didn't hallucinate anything; it was given bad input and reasoned correctly from it.

That's the value a trace adds over a log line: not just "what happened" but "what specific data moved between which specific steps," which turns a wrong-answer bug report into a one-argument fix (passing delivery date instead of purchase date to the policy tool) instead of an open-ended re-investigation.

The cost of instrumenting: overhead and sampling

Tracing isn't free, on two axes that matter differently at different scales.

Latency overhead from the tracing calls themselves (serializing span attributes, exporting to a collector) is usually small relative to an LLM call's own latency -- a few milliseconds against a call that takes hundreds to thousands -- so it's rarely the bottleneck for a single request. Where it adds up is at high request volume, where the export path (network calls to a collector, or to a purpose-built platform's ingestion API) becomes its own capacity-planning concern, particularly if spans are exported synchronously rather than batched.

Token and cost overhead is the sharper trade-off. Capturing full prompts and responses (rather than truncated or hashed versions) means storing -- and in the hosted case, transmitting to a third party -- the same volume of text the LLM itself processed, which at scale is a real storage/egress cost, separate from the LLM call's own token cost. Two practical mitigations cover most cases: sample at less than 100% for high-volume, low-value traffic (trace every Nth request fully, or trace based on an outlier heuristic like unusually high latency or a non-default temperature) while always tracing 100% of requests that end in an error or a user-reported issue; and truncate or redact captured prompt/response content for spans that clear a size threshold, keeping enough for debugging shape without paying to store every token of every run.

The failure mode to avoid is the inverse of both: 100%-sampling full content by default because it's the path of least resistance during initial setup, then discovering the storage or vendor bill months later and retrofitting sampling under pressure, at which point the traces from the period that would have been most useful to look back on were never captured cheaply enough to keep.

Building your own: what's actually involved

"Build it on OpenTelemetry" understates the work if left at that level of abstraction. Concretely, a minimal custom tracer needs: an OTel SDK initialized with a service name and a configured exporter (OTLP to whatever backend -- Jaeger, Tempo, an existing APM vendor's OTel-compatible ingestion -- the team already runs); a thin wrapper function around every LLM call site that opens a span, sets the attributes listed earlier (prompt template id, resolved variables, response, token counts, cost, model/provider), and closes the span on completion or error; the same wrapper pattern around tool calls; and propagation of the trace/span context across async boundaries (a plan step that dispatches to a background job, or a sub-agent call) so the resulting trace tree doesn't fragment into disconnected pieces -- this last part is the detail most homegrown tracers get wrong first, because it only becomes visible once a run has enough async fan-out for a naive setup to lose the parent-child link.

None of this is exotic engineering -- most of it is the standard OTel instrumentation pattern applied to a new call type -- but it is real work that a purpose-built platform's SDK does out of the box, which is exactly the trade this section of the article is about: build-your-own wins on unification with existing infrastructure and loses on time-to-first-usable-trace.