Why architecture matters here
Why does agent evaluation need its own architecture instead of assertEquals? Because correctness is distributional: an agent is acceptable when a high fraction of runs land within tolerance, so the unit of testing is a scored sample, not a single assertion. Because the path matters independently of the answer: an agent that answers a refund question correctly without checking the order got lucky and will hallucinate next week — trajectory evaluation exists to catch the right answer reached wrongly, which endpoint testing structurally cannot. And because everything is coupled to everything: a sub-agent description edit shifts routing; a tool docstring tweak changes argument patterns; a model version bump moves every distribution at once. Without a regression suite over trajectories, each change is a bet placed blind.
The economics are as important as the correctness. Prompt engineering without evals converges on superstition — changes accrete because 'it seemed better', nobody dares delete anything, and the instruction grows barnacles. With evals, prompt work becomes engineering: propose, run the suite, read the deltas, keep or revert. Model migrations — the recurring event of this era — become a weekend instead of a quarter: run the suite against the new model, triage the failures, fix, ship. Teams that skipped eval infrastructure discover its absence exactly when a deprecation deadline forces a migration with no safety net.
There is also an organizational function: eval cases are executable specifications. 'The agent must verify ownership before revealing order details' is a sentence in a doc until it is an eval case asserting the lookup-before-disclosure trajectory; then it is a gate no future refactor can silently violate. The eval set becomes where product, security, and engineering encode what the agent must do — reviewable, versioned, and enforced on every merge.
The architecture: every piece explained
Top row: from conversation to test. An eval set is a versioned file of cases grouped around a capability (refunds, order lookup, escalation). Each eval case holds a conversation: user turns, optional initial session state (the fixture — user identity, entitlements), the expected tool trajectory (ordered tool calls with expected arguments, captured or hand-authored), and the reference response per turn. The cheapest way to author one is to have a good (or instructively bad) conversation in adk web and save it as a case, then edit expectations — curation from reality beats invention. Runner replay executes each case against the current agent tree with the fixture state: real callbacks, real routing, tools either live or stubbed (a deliberate choice — live catches integration drift, stubs isolate agent logic and run in CI without credentials). The output is the actual trajectory: the full event stream of the replay.
Middle row: scoring the two axes. Tool trajectory scoring compares expected vs actual call sequences — tool names, argument matching (exact or per-field tolerance), order — yielding a score rather than a boolean, because an extra harmless lookup differs from a skipped ownership check. Response matching scores final text against the reference with similarity metrics (ROUGE and cousins) — crude but fast and calibration-free, good for catching wholesale regressions. LLM-as-judge covers what similarity can't: a rubric ('accurate per the tool results? polite? no policy violations? no invented facts?') scored by a strong model against the transcript and evidence — the axis that scales to open-ended quality, at the cost of judge calibration (fixed judge model/version, spot-audited against human labels). Thresholds convert scores to verdicts per case and per suite: trajectory ≥ 0.9, response ≥ threshold, judge ≥ rubric floor — tuned so the suite fails on regressions, not on phrasing variance.
Bottom rows: the loop closes. adk web / CLI is the authoring and debugging surface — run a set, inspect a failing case's replay event-by-event next to the expectation. CI integration (pytest wrappers around the eval runner) makes every PR that touches instructions, tools, or agent wiring run the relevant suites; model or prompt changes get the full suite plus N-run sampling for variance-sensitive cases. The ops strip: coverage maps (which capabilities have cases), drift dashboards (nightly runs against production model versions), and eval-cost budgeting — a full suite with judge scoring costs real tokens, so tiering (smoke set per PR, full set nightly) keeps signal per dollar honest.
End-to-end flow
Walk a change through the pipeline. Monday: support asks that the agent stop offering refunds proactively — policy now requires the user to ask. An engineer edits the refund agent's instruction. The PR touches an instruction file, so CI runs the refund suite (32 cases) and the routing smoke set (20 cases across all capabilities).
Results: refund suite 30/32. The two failures are instructive. Case 17 ('item arrived broken — what are my options?') expected the agent to mention refund among options; the new instruction made it reticent — the response-match score fell below threshold. Product confirms options-listing should still include refunds; the engineer tightens the instruction distinction (offer ≠ enumerate) and case 17 passes. Case 24 fails on trajectory: the agent now calls lookup_order twice — the instruction edit made it re-verify before acting. Harmless but wasteful; the trajectory diff made the cost visible; a one-line instruction fix removes it. The routing smoke set: 20/20 — evidence the edit didn't leak into triage behavior. Merge.
Wednesday: the platform team trials the new model version. Full suite, all capabilities, 5 runs per case for variance: 94% → 91% aggregate, with failures clustered in the escalation suite — the new model transfers to human handoff less eagerly on ambiguous fraud signals. That cluster analysis is the value: not 'the model is worse' but 'this specific safety behavior regressed'. The fix is a sharpened escalation criterion in the security agent's instruction; re-run, 95%, migrate. Friday: the nightly drift run — same suite, production model — dips on one case where a tool's live sandbox changed its response shape; the alert fires before any user notices, and the tool wrapper gains a normalization. Three different change vectors — prompt, model, dependency — one safety net, and at no point did anyone argue from anecdotes.