Why architecture matters here
The case for agent-specific observability starts with the failure taxonomy. Agents fail in ways that are invisible to RED metrics: a routing regression sends 30% of billing questions to the shipping specialist — every request succeeds, latency is fine, and users get confidently unhelpful answers; a prompt tweak makes the critic agent too lenient and quality decays with zero errors; compaction starts dropping commitments and the agent breaks promises made forty turns ago. Each is detectable — but only by instruments aimed at behavior: routing-accuracy sampling, eval-score trends, escalation-rate shifts, commitment-consistency checks on replayed sessions. Teams that monitor agents like web services learn about behavioral regressions from customers.
Second, the debugging workflow is different in kind. A stack trace localizes a crash; an agent mis-answer localizes nothing — the cause might be retrieval (wrong document), context (truth lost to windowing), tools (misleading result shape), routing (wrong specialist), or the model (plain hallucination). The only tractable method is time travel: replay the session's event log, watch state fold and context assemble turn by turn, and find the step where the world model diverged. This is why the event log's completeness is the load-bearing property — every tool argument, every state delta, every transfer with its author — and why teams that log 'the response' but not 'the assembled request' find themselves unable to answer the only question that matters: what did the model actually see?
Third, economics are a first-class signal, not billing hygiene. Token consumption per resolved task is simultaneously a cost metric, a performance metric (context bloat shows here first), and a security metric (injection-induced loops and exfiltration attempts spike it). A cost anomaly detector with per-session granularity catches entire failure classes — degenerate transfer ping-pong, tool retry storms, context-flooding results — hours before their behavioral symptoms accumulate. The mature posture treats tokens the way SRE treats saturation: a golden signal with budgets, attribution, and alerts.
The architecture: every piece explained
Top row: the four lenses. The event log is ground truth: every session's ordered events (messages, function calls/results, state deltas, transfers, control actions) with authors and invocation ids — queryable for audit ('all sessions where issue_refund executed above X'), replayable for debugging, mineable for evals. OTel traces give structure per invocation: a root span per turn, child spans per model call (attributes: model version, input/output token counts, finish reason) and per tool execution (tool name, duration, result size, error kind); ADK's instrumentation emits these natively, and they flow to any OTel backend. Token metrics aggregate the span attributes into the economic view: tokens and cost by agent, by tool, by session percentile, by version — with cost-per-resolved-task as the north-star ratio. Quality signals come from three sources: scheduled eval runs (score trends per capability), online sampling (routing-accuracy labeling queues), and behavioral counters (escalation rate, human-handoff rate, guardrail-block rate) that proxy quality between eval runs.
Middle row: making it usable. Structured logging ties the layers: every log line, span, and event carries (app, agent, session id, invocation id), so a cost anomaly pivots to its trace, the trace to its session, the session to replay — one correlation model, enforced by the harness so nobody forgets a field. Dashboards split by audience: the on-call view (turn latency, error and guardrail-block rates, token burn, queue depths), the product view (resolution rate, escalations, satisfaction proxies), the platform view (per-version comparisons during canaries). Session replay is the debugger: the dev UI (or programmatic replay) steps through events with assembled-context inspection — the agent equivalent of stepping through frames. Alerting spans both natures: infrastructure alerts (latency, errors, saturation) plus behavioral alerts (eval-score drop on the nightly run, routing-accuracy sample below threshold, cost-per-session outliers, guardrail-block spikes) — the second set catches what the first structurally cannot.
Bottom rows: the sensors and the loop. Callback instrumentation: before/after hooks are where custom telemetry lives — a before_model hook stamps context-size histograms by component (the context-engineering dashboard), an after_tool hook records result sizes and sanitization hits, guardrail callbacks emit tagged block events. Production mining closes the quality loop: failed, escalated, low-rated, and anomalous sessions flow into a review queue; reviewed sessions become eval cases; the eval suite's coverage grows toward the traffic's real distribution. The ops strip holds the disciplines: PII-safe telemetry (redaction before export — prompts and results carry user data; span attributes and logs must not), sampling strategy (full events always — they're the record; verbose traces sampled), and the correlation contract enforced in CI.
End-to-end flow
Run an incident and a quality investigation through the stack. Tuesday 15:40 — the cost-anomaly alert fires: session p99 token burn tripled in an hour, isolated to the support agent, version unchanged. Pivot: top sessions by burn → traces show turns with 12+ tool spans, all search_kb, near-identical arguments. Pivot to replay: at turn 3, a KB tool result began returning an error-shaped payload the model read as 'try again' — a KB-service deploy at 15:20 changed the empty-result shape from {results: []} to {error: 'no matches'}, and the model retried in good faith, per session, forever. Containment: after_tool normalization patch mapping the new shape (15 minutes); the KB team reverts; the retry-loop pattern gets a budget-guardrail eval case; the alert-to-diagnosis time was 22 minutes because every layer pivoted to the next by ids that were always there.
The quality investigation is slower-burn. The weekly product review notes escalation rate creeping up 2 points over three weeks, no deploys in the window — but the nightly eval trend shows the refund capability's judge scores easing down in the same period. Mining pulls twenty recent escalated refund sessions into review; replay shows a pattern: conversations exceeding the compaction threshold escalate at twice the rate, and the compaction summaries — spot-checked in replay — sometimes drop the refund method the user specified early. It's the summarizer prompt: a model-side update (the pinned model had a minor revision in its serving stack) changed summary style just enough. Fixes ship in layers: refund method moves to structured state (immune to summarization by construction — the durable fix), the summarizer prompt gets a must-preserve list tightening, and two compaction eval cases are minted from the mined sessions. Six days later the trend lines recover; the review closes with the numbers attached.
The quarter's audit request shows the third face: compliance asks for every session where a refund above ₹10k executed, with approver identity. It is one query over the event log (tool-execution events, argument filter, joined approval events) — twenty minutes to a signed report, because the runtime's native record was designed as evidence, and the observability stack never had to bolt an audit trail onto a system that lacked one.