'The agent failed' is not a debuggable statement
An agent that plans, calls three tools, reflects, and replans once before returning an answer has made somewhere between five and fifteen separate model or tool invocations by the time it responds. When the final answer is wrong, "the agent failed" describes an entire tree of decisions, any one of which could be the actual root cause -- a bad plan, a tool called with wrong arguments, a tool that returned correct data the model misread, or a reflection step that approved a flawed intermediate result. Without visibility into that tree, debugging degenerates into re-running the same request and hoping the failure reproduces somewhere visible.
This is an operational and debugging topic, not a security one -- distinct from a vulnerability like confused-deputy attacks, which is about an agent being tricked into misusing its own authority. Tracing doesn't prevent an agent from making a wrong decision; it makes the decision-making visible enough that a wrong one can be found and fixed instead of guessed at.
A span per call, not a log line per run
The unit of observability for an agent is the span -- the same primitive distributed tracing uses for a microservice call graph -- not a single log line summarizing the whole run. Every LLM call gets its own span: the prompt sent, the response received, latency, token counts, and cost. Every tool call gets its own span: which tool, what arguments, what it returned, whether it errored. A run that made twelve calls produces twelve spans, not one line saying "agent completed in 4.2s."
The reason granularity at this level matters specifically for agents, more than for a typical service call graph: an agent's next action depends on the previous one's output in a way that's chosen by the model rather than fixed in code, so the interesting failures are frequently about what the model did with a span's output, not just whether that span itself succeeded. A tool call that returned successfully but with data the model then misinterpreted is invisible to a health check and invisible to a coarse "did it error" summary -- it's only visible if the span capturing the model's next reasoning step is linked to the span whose output it was reasoning about.
Parent-child structure across the plan-execute-reflect loop
A flat list of spans in call order is better than nothing but throws away the structure that actually explains a failure: which spans were part of the same planning phase, which tool calls executed which plan step, and which reflection span evaluated which execution's output. Structuring spans as a tree -- a root span for the overall task, child spans for plan/execute/reflect/replan phases (see agent planner architecture for the loop this maps onto), and grandchild spans for individual tool and model calls within each phase -- preserves the causal structure a flat log discards.
task: "triage bug report and propose fix" [root span, 4.2s, $0.14]
├── plan [span, 1.1s]
│ └── llm_call: planner [span, 1.1s, 1900 tok]
├── execute: step 1 (read report) [span, 0.3s]
│ └── tool_call: read_file [span, 0.2s, ok]
├── execute: step 4 (diagnose) [span, 1.6s]
│ ├── tool_call: run_tests [span, 0.9s, ok]
│ └── llm_call: diagnose [span, 0.7s, 1200 tok]
├── reflect [span, 0.4s]
│ └── llm_call: reflect [span, 0.4s, "matches report: yes"]
└── execute: step 5 (draft fix) [span, 0.6s]
└── llm_call: draft_patch [span, 0.6s, 900 tok]With this structure, "why did the agent propose the wrong fix" becomes a walk down one branch -- was the plan reasonable (check the plan span), did diagnose get correct tool output (check its tool-call child), did diagnose reason correctly from that output (read the llm_call child's prompt/response pair) -- rather than a re-run-and-hope.
What to actually capture per span
Latency and a status code are the minimum any tracing system captures and are almost never enough on their own for an agent. The fields that make a span actually debuggable: the full prompt sent to the model for that call (not just "called the model"), the full response, token counts split input/output, per-call cost, and for tool spans the exact arguments passed and the exact value returned -- not a truncated summary, the actual payload, because the misread-tool-output failure mode from the first section is only diagnosable if you can see precisely what the model saw.
This is more data than a typical service trace captures, and it's worth it specifically because the debugging question for an agent is usually "what did the model see and how did it reason from that," which a status code can never answer. Cost tracking belongs at the span level too, not just aggregated per run -- a run that cost $4 when it should have cost $0.20 is explained by finding which specific span burned the tokens, which ties directly into agent cost optimization: you can't optimize what you can't attribute to a specific decision point.
Debugging by walking the tree, not re-running
The practical payoff of the structure above is that a failed run doesn't need to be reproduced to be debugged -- the trace tree from the actual failing run already contains the evidence. Start at the root, check whether the final output's problem traces to a specific phase (compare what reflection approved against what execution actually returned -- a mismatch there is a reflection bug, not an execution bug), then descend into that phase's children until you reach the specific span where reasoning diverged from what the input actually supported.
This matters more the less deterministic the system is. Re-running an agent on the same input can produce a different trajectory entirely, since the model's next action depends on its own prior output in a chain that isn't guaranteed to repeat identically -- which means a bug that doesn't reproduce on re-run isn't fixed, it's just hiding in a trace nobody captured the first time. Capturing the trace from every run, not just the ones that fail obviously, is what makes intermittent failures debuggable instead of anecdotal.
Trace an agent the way a distributed system traces a service call graph -- a span per LLM call and per tool call, structured as a parent-child tree that mirrors the plan-execute-reflect loop, capturing full prompts, responses, and tool payloads rather than status codes alone. The payoff is debugging a specific failed run by walking its actual trace tree instead of re-running a non-deterministic system and hoping the failure repeats somewhere visible.