Why architecture matters here
Classic APM assumes a deterministic handler: same input, same code path, roughly the same latency. An LLM agent violates every one of those assumptions. The model may answer directly on Monday and call three tools on Tuesday for the same prompt, because a session carried different state or the model sampled a different plan. Latency is not a distribution around a mean; it is a sum of a variable number of sequential model calls, each with its own time-to-first-token. Cost is a first-class production metric — a single runaway loop can spend more in an hour than the rest of the fleet does in a day — and no traditional monitoring stack has a column for it.
Attribution is the second forcing function. When an answer is wrong, the defect could live in the system prompt, in a tool's output that the model faithfully repeated, in stale session state, in the orchestration that picked the wrong sub-agent, or in the model itself. Only a trace that records the exact prompt assembled for each call, the tool arguments and results, and the ordering of events can distinguish these — and only if that trace is captured in production, because agent failures are famously unreproducible in staging where the session history differs.
Finally, agents change faster than services. Prompt edits, tool description tweaks, and model version bumps ship weekly, and each one can silently shift behavior across every conversation. Without per-version telemetry and a stream of scored production traces, teams discover regressions from user complaints. The observability architecture is what turns 'the agent feels worse' into a diff between two dashboards.
The architecture: every piece explained
The span tree. The root span wraps the Runner invocation: one user message in, one final response out. Beneath it, each agent turn gets a child span — important in multi-agent trees where a coordinator delegates to specialists — and beneath those, two span kinds carry the real payload. LLM call spans record the model id, request token count, response token count, finish reason, time-to-first-token, and a digest (not the full text) of the assembled prompt. Tool spans record the tool name, a redacted argument digest, result size, and latency. Because ADK executes the loop, the instrumentation can live in one place — a wrapper around the model client and the tool executor — rather than in every agent.
Callbacks as tap points. ADK Java's callback interfaces are where cross-cutting telemetry belongs. beforeModel sees the fully assembled request: record prompt-section sizes (system, history, tool schemas) so you can explain context growth. afterModel sees usage metadata: increment token counters tagged by agent and model version. beforeTool is the redaction and policy point — hash or drop sensitive arguments before they reach a span. afterTool classifies results into success, business error, and system error, feeding the tool error-rate metric. Callbacks must be exception-safe: telemetry that throws should never kill the turn.
Session events as ground truth. Every turn appends events — user message, model response, tool call, tool result, state delta — to the session store. This log is transactional with the agent's behavior in a way exporters are not: if OTLP batches drop under backpressure, the session record still holds the complete conversation. Treat traces as the fast index and session events as the durable record; store the trace id in the event so either can find the other.
Metrics and logs. Micrometer (or the OTel metrics API) carries the aggregates: token counters by agent and direction, latency histograms per model call and per invocation, tool error rates, and — the agent-specific one — a histogram of loop iterations per invocation. Structured logs carry trace id, session id, and invocation id on every line so a support engineer can pivot from a user complaint to the exact trace in two queries.
End-to-end flow
A user message arrives and the Runner opens the root span, stamping session id and app version as attributes. The root agent's turn span opens beneath it. beforeModel fires: the callback records that the assembled request is 6,200 tokens — 1,100 of system prompt, 3,800 of history, 1,300 of tool schemas — and checks it against the invocation's token budget. The LLM span opens, the call streams, and time-to-first-token lands at 640 ms. The model returns a tool call rather than an answer; afterModel records 6,200 input and 45 output tokens and the finish reason tool_call.
beforeTool fires for lookupOrder: the order id is kept, the customer email is hashed, and the policy check passes. The tool span wraps the execution — 210 ms against the order service, whose own spans join the same trace via context propagation, so the agent trace and the downstream microservice trace are one tree. afterTool marks the result a success at 2.1 KB, and the loop re-enters: a second model call, this time 6,900 tokens in, produces the final answer. The root span closes with attributes for total tokens (13,300), model calls (2), tool calls (1), and loop iterations (2).
Asynchronously, the exporter batches spans to the backend; the metrics pipeline flushes counters; the session store now holds seven new events carrying the trace id. A sampler watches root spans and forwards 2% of successful invocations — and 100% of those with errors, budget overruns, or five-plus iterations — into the eval pipeline, where an LLM judge scores groundedness and policy compliance against the recorded tool results. Scores flow back as metrics tagged by prompt version, closing the loop: tomorrow's prompt edit will show up as a scored delta, not a vibe.