Why architecture matters here
Architecture matters here because multi-agent systems are distributed systems, and distributed systems are only as debuggable as their tracing. The whole appeal of A2A architectures — composing specialists, fanning work out, delegating across teams and networks — is exactly what makes them opaque without tracing. Every delegation is a boundary across which context is normally lost, and every lost boundary is a place where a latency spike or an error can hide. Tracing is the connective tissue that turns a pile of independent agents into a system you can reason about.
The problem is genuinely hard because the trace must survive boundaries the individual agents do not control. An orchestrator and a specialist agent may be separate services, written by separate teams, deployed independently, possibly in different runtimes. For their spans to land in the same trace, they must agree on a propagation format and both honor it. If the orchestrator injects W3C Trace Context headers but the specialist ignores them and starts a fresh trace, you get two disconnected traces for what was logically one task, and the causal link — the single most valuable thing tracing provides — is gone.
Consider a concrete debugging scenario to feel the value. A user reports that a research task took 40 seconds. Without tracing, you check the orchestrator's logs (it was waiting on a delegate), then the delegate's logs (it was waiting on a tool), then the tool's logs — three separate investigations correlated by hand. With a trace, you open one waterfall and immediately see that 34 of the 40 seconds were spent in a single web-search tool call inside the third sub-agent. The answer that took an hour of log archaeology is now a glance, because the trace already encodes the causal structure.
There is a cost dimension unique to agent systems that makes tracing even more valuable than in ordinary microservices. Every LLM call has a token count and a dollar cost, and in a fan-out architecture a single user request can trigger dozens of model invocations across many agents. Attaching token and cost attributes to each LLM span means the trace does not just show where time went but where money went — a per-trace cost breakdown that is essential for understanding and controlling the economics of a multi-agent product.
The payoff of getting tracing right is that observability composes the same way the agents do. Add a new specialist agent, and as long as it extracts and propagates trace context and emits standard spans, it appears automatically in every trace that touches it — no bespoke correlation logic, no per-agent dashboards. The service map, the latency waterfall, the error attribution, and the cost breakdown all extend to the new agent for free, which is what lets a multi-agent system grow without becoming unobservable.
The architecture: every piece explained
Top row: the propagation path. A user task opens a root span in the first agent, establishing the trace id that everything downstream will share. When that agent delegates to another over A2A, context injection writes the trace id and the current span id into the outgoing request — conventionally as W3C Trace Context headers (traceparent/tracestate). The remote agent extracts that context on receipt and continues the same trace, opening its work as child spans rather than starting fresh. Within any agent, tool and LLM spans nest under that agent's span, one child per tool call or model invocation.
Middle row: what makes spans useful. Span attributes are the structured tags on each span — the agent name, the model used, the token counts, the estimated cost, the tool name, the input and output sizes. These are what let you filter and aggregate: 'show me the slowest spans for the pricing agent' or 'sum tokens across this trace.' The sampling decision determines whether a given trace is recorded at all, and critically that decision must be made once at the root and propagated, so either the whole trace is kept or the whole trace is dropped — never half of it. The exporter batches finished spans and ships them out of the agent process asynchronously so tracing never blocks the request path.
Bottom row: assembly and viewing. The collector receives spans from every agent and groups them by trace id, reconstructing the tree from each span's parent pointer. The trace backend presents the assembled trace as a waterfall (a timeline of nested spans showing exactly where time went) and often a service map (which agents call which). Because the collector assembles by trace id regardless of which agent emitted a span, agents can export independently and asynchronously and the backend still stitches them into one coherent picture.
The architectural crux is that the trace context is the only thing that must cross every boundary, and it is tiny — just a couple of identifiers and flags. Everything else (attributes, spans, sampling machinery) is local to each agent and exported independently. This is what makes tracing scale across independently deployed agents: the shared contract is a small header format, not a shared database or a synchronous call. Each agent does its own instrumentation and export; the collector does the assembly.
The ops strip is the payoff surface. End-to-end latency is the root span's duration — the number the user actually feels. Per-agent time, read off the waterfall, shows which agent or tool dominates. Error attribution uses the span where an error was first recorded to pin the origin, distinguishing where a failure started from where it surfaced. Token and cost per trace, summed from the LLM spans' attributes, give the economic view. Together they turn a multi-agent black box into a measured, attributable system.
End-to-end flow
Trace a research task through a three-agent system. The user's request hits the orchestrator, which opens the root span (trace id T, span id S0). The orchestrator decides it needs a search specialist and a summarizer, so it makes an A2A call to the search agent, injecting T and S0 into the request headers as the trace context.
The search agent receives the call and extracts the context. Instead of starting a new trace, it opens a span S1 with trace id T and parent S0 — now part of the same trace. Inside its turn it calls a web-search tool (child span S2 under S1, tagged with the query and result count) and an LLM to rank results (child span S3 under S1, tagged with the model, prompt tokens, completion tokens, and computed cost). It finishes, exports S1, S2, and S3 asynchronously, and returns its result to the orchestrator.
The orchestrator then calls the summarizer agent, again injecting T and the orchestrator's span id. The summarizer opens span S4 under S0, makes its own LLM call (span S5 under S4 with its own token and cost attributes), finishes, and exports. The orchestrator collects both delegate results, produces the final answer, closes the root span S0, and returns to the user. Every span emitted by every agent carried the same trace id T and a correct parent pointer.
Now the collector does its job. Spans S0 through S5 arrive from three different agent processes at slightly different times, asynchronously. The collector groups them all by trace id T and reconstructs the tree from the parent pointers: S0 is the root, S1 and S4 are its children, S2 and S3 hang under S1, S5 hangs under S4. The backend renders this as a waterfall showing the full task timeline. If the whole task took 8 seconds and S2 (the web search) took 6 of them, that is instantly visible — the slow tool is named and located without any manual log correlation.
Step back and count what propagation bought across that task. Three independently deployed agents, five tool and model calls, and one user request all appear as a single connected tree because the trace context crossed every A2A hop. Latency is attributable to a specific span, errors would be pinned to the span that first recorded them, and total cost is the sum of the token-tagged LLM spans S3 and S5. Had any agent failed to propagate the context, its spans would have formed an orphan trace and that entire branch of the task would have vanished from the picture. The whole value — one coherent, attributable, costed view of a distributed task — rests on the discipline of injecting and extracting that small context on every hop.