An agent’s behaviour is a distribution, not a return value, so you cannot unit-test it with assertEquals. The Agent Development Kit answers with an evaluation harness: a file format for recording what an agent should do, a runner that replays those recordings against your live agent, and a set of scorers that grade the result on two axes at once. This piece is the mechanics — not why evaluation matters (that is its own architecture story), but the concrete tooling you actually touch: the shape of a .test.json versus a .evalset.json, exactly what one eval case captures, the built-in metric keys and their default thresholds, and the three ways to press play — the adk eval CLI, AgentEvaluator.evaluate inside pytest, and the adk web UI — ending with how to turn a passing suite into a CI gate. Everything below is copy-pasteable and version-checked against current ADK.
Two artifacts: a test file and an eval set
ADK stores expected behaviour in two file kinds that share one schema but serve different jobs. A test file (*.test.json) is the lightweight unit — conventionally a small, single-session scenario you keep next to the code it exercises and run in pytest. An eval set (*.evalset.json) is the heavier integration artifact: many eval cases grouped around a capability (refunds, order lookup, escalation), typically multi-turn, and usually authored through the web UI by saving real sessions.
The practical division is scale and provenance. Test files are hand-sized and hand-authored — you write a couple by hand to pin one behaviour and check them into the same folder as the agent. Eval sets are curated collections, often dozens of cases, versioned as a suite and owned jointly by product and engineering. Both are just JSON conforming to ADK’s EvalSet schema, so the runner treats them identically; the naming convention (.test.json vs .evalset.json) signals intent to humans, not to the tool. Start with test files while a capability is small, graduate to an eval set once you have enough cases that a shared, browsable collection earns its keep.
What a single eval case captures
An eval case is a recorded expectation about one conversation. Four things make it up. First, the query — the user turn(s) that drive the agent, stored as user_content with a parts array and role: "user". Second, the expected tool trajectory — the ordered list of tool calls the agent should make to satisfy the query, stored under intermediate_data.tool_uses, each entry a name plus an args object. Third, the reference response — the expected final answer, stored as final_response with its own parts and role: "model". Fourth, the initial session state — the fixture the replay starts from, stored in session_input as app_name, user_id, and a state dictionary.
That fourth field is what makes cases realistic without external setup. A refund case can begin from {"authenticated": true, "tier": "gold"} so the agent is exercised as a logged-in gold-tier user, no login flow required. Multi-turn cases string several invocations together in the conversation array; each invocation carries its own query, expected trajectory, and reference, so the harness scores a whole dialogue turn by turn rather than only a final answer.
The EvalSet JSON, up close
Here is a trimmed but faithful eval set with one case. The nesting is the thing to internalize: eval_cases → a conversation of invocations → each invocation’s user_content, final_response, and intermediate_data.tool_uses, with session_input hanging off the case as its fixture.
{
"eval_set_id": "refund_flow",
"name": "Refund capability",
"description": "Broken-item and mistaken-order refund paths.",
"eval_cases": [
{
"eval_id": "broken_item_refund",
"conversation": [
{
"invocation_id": "inv-1",
"user_content": {
"role": "user",
"parts": [{"text": "My blender arrived broken, order A-4471."}]
},
"intermediate_data": {
"tool_uses": [
{"name": "lookup_order", "args": {"order_id": "A-4471"}},
{"name": "issue_refund", "args": {"order_id": "A-4471"}}
],
"intermediate_responses": []
},
"final_response": {
"role": "model",
"parts": [{"text": "I've verified order A-4471 and issued a refund."}]
}
}
],
"session_input": {
"app_name": "customer_support",
"user_id": "user-9",
"state": {"authenticated": true, "tier": "gold"}
}
}
]
}You rarely type this by hand end-to-end — the web UI emits it from a recorded session — but you will hand-edit it to tighten expectations, so knowing which field is which is not optional.
The lightweight test-file shape
Because the full EvalSet schema is verbose, ADK has long supported a flatter, human-friendly test-file form for the common single-turn case: a JSON array where each entry pairs a query with an expected_tool_use list and a reference answer. It is the fastest way to pin a simple tool-calling behaviour without the invocation scaffolding.
[
{
"query": "What's the weather in Paris?",
"expected_tool_use": [
{"tool_name": "get_weather", "tool_input": {"city": "Paris"}}
],
"reference": "It's currently clear and about 18 degrees in Paris."
}
]Note the field names differ from the EvalSet form — here a tool call is tool_name/tool_input, whereas inside intermediate_data.tool_uses it is name/args. Do not cross them; that mismatch is the classic sign of a hand-copied file that will silently fail to parse the trajectory. New work trends toward the EvalSet schema (it is what the UI produces and what multi-turn cases need), but the flat form remains a pragmatic choice for a quick, single-turn regression check.
Two dimensions: did it act right, did it answer right
The harness scores every replay on two independent axes, and keeping them separate is the central design idea. The first is tool-trajectory match: did the agent call the expected tools, with the expected arguments, in the expected order? The second is response quality/match: does the final answer say what the reference says?
They are separate because either can pass while the other fails, and each failure means something different. An agent that produces the right words with the wrong trajectory — answering a refund question without ever calling lookup_order — got lucky and will hallucinate on the next input; only trajectory scoring catches it. An agent with a perfect trajectory but a garbled final message did the work and then fumbled the summary; only response scoring catches that. Endpoint-only testing collapses these into one and misses the first entirely, which is precisely why agent evaluation needs a purpose-built harness rather than a string comparison. Every built-in metric slots into one of these two axes, and a case’s verdict is the conjunction: it passes only when both the trajectory and the response clear their thresholds.
The built-in metrics
ADK ships a set of named metrics you compose per suite. The two workhorses are cheap and deterministic; the rest bring an LLM judge for what similarity cannot measure.
| Metric key | Axis | What it measures | Default |
|---|---|---|---|
tool_trajectory_avg_score | Trajectory | Exact match of the tool-call sequence (names + args + order), averaged over steps | 1.0 |
response_match_score | Response | ROUGE-1 similarity of the final answer to the reference | 0.8 |
final_response_match_v2 | Response | LLM-judged semantic equivalence to the reference | rubric floor |
rubric_based_final_response_quality_v1 | Response | LLM-judged quality against a rubric, no reference needed | rubric floor |
hallucinations_v1 | Response | Groundedness of the answer in tool results / context | rubric floor |
safety_v1 | Response | Harmlessness / policy compliance of the response | rubric floor |
Reach for tool_trajectory_avg_score and response_match_score first: they are free, fast, and calibration-free, so they belong on every smoke-test PR. Because response_match_score is ROUGE-based it rewards lexical overlap, so keep references phrased the way the agent actually answers or the score punishes harmless wording changes. When you need to grade meaning, open-ended quality, groundedness, or safety — things ROUGE is blind to — add the LLM-judged _v2/_v1 metrics, and pin the judge model/version so scores are comparable across runs.
Thresholds and test_config.json
A metric produces a number; a threshold turns it into a verdict. Thresholds live in a test_config.json placed beside your eval files, under a criteria object keyed by metric name. Any metric you list is active; anything you omit is not scored. When no config is present, ADK applies the built-in defaults — 1.0 for tool_trajectory_avg_score and 0.8 for response_match_score.
{
"criteria": {
"tool_trajectory_avg_score": 1.0,
"response_match_score": 0.7
}
}The tuning instinct that keeps a suite useful: hold trajectory strict and let response breathe. A perfect 1.0 on trajectory is reasonable to demand — the agent either called the right tools in the right order or it did not, and slippage there is a real regression. Response match is where phrasing variance lives, so a threshold in the 0.6–0.8 band usually catches wholesale answer regressions without flagging every reworded sentence. Set it too high and the suite cries wolf on cosmetic diffs until people stop reading it; too low and a genuinely wrong answer sails through. Tune against a handful of known-good and known-bad runs, not by guessing.
Authoring cases: capture, then curate
The cheapest good eval case is a real conversation you liked. In adk web, drive the agent through a scenario — a clean refund, an instructively broken one — then open the Eval tab and save that session into an eval set. ADK records the actual queries, the tool calls the agent made, and the responses it produced, writing them into the EvalSet schema for you. You then curate: trim the trajectory to the calls that matter, fix up an args value, replace a mediocre final answer with the reference you actually want, and set the initial state.
Curation-from-reality beats invention on every axis. The captured trajectory is already syntactically valid and uses real tool names and argument shapes, so you are editing a working artifact rather than guessing field names into a blank file. It also captures behaviours you would not think to write — the exact argument the model tends to pass, the intermediate step it takes — which is where regressions actually hide. Treat saved sessions as raw material: capture generously, then keep only the expectations you are willing to defend as a specification, because every field you leave in becomes a thing future changes must satisfy.
Running evals: the CLI
The command-line runner is the fastest path from a file to a verdict, and the natural fit for scripts and CI. Point adk eval at your agent module and an eval file:
# run an entire eval set
adk eval path/to/agent path/to/refund.evalset.json
# supply thresholds and print per-case detail
adk eval path/to/agent path/to/refund.evalset.json \
--config_file_path=path/to/test_config.json \
--print_detailed_results
# run only specific cases by eval_id
adk eval path/to/agent path/to/refund.evalset.json:broken_item_refund,mistaken_orderThe first positional argument is the agent module path; the second is the eval file. --config_file_path supplies the test_config.json criteria (otherwise defaults apply), and --print_detailed_results expands the output from a pass/fail summary to per-case, per-metric scores — indispensable when triaging a failure, because it shows you which axis dipped and by how much. Appending :id1,id2 to the file path runs a subset, which is how you iterate on one stubborn case without paying for the whole suite each loop.
Running evals: pytest
For CI and for keeping evals next to your other tests, ADK exposes AgentEvaluator.evaluate, an async helper you call from a normal pytest test. It runs the eval file, applies the thresholds, and raises an assertion failure if any case falls short — so a failing eval is just a failing test, with no bespoke reporting to wire up.
import pytest
from google.adk.evaluation.agent_evaluator import AgentEvaluator
@pytest.mark.asyncio
async def test_refund_flow():
await AgentEvaluator.evaluate(
agent_module="customer_support.agent",
eval_dataset_file_path_or_dir="tests/evals/refund.evalset.json",
num_runs=2,
)agent_module is the import path to the module that exposes your root agent. eval_dataset_file_path_or_dir accepts a single file or a directory — pass a folder and it runs every eval file inside, which is how you gate a whole capability with one test. num_runs replays each case N times to sample variance: agents are stochastic, so a case that passes once may be a coin flip, and averaging over a few runs turns a flaky gate into a meaningful one. A sibling test_config.json in the same directory supplies the thresholds automatically.
Running evals: the web UI
The adk web UI is the authoring and debugging surface, and it closes the loop the other two runners open. Beyond capturing sessions into eval sets, its Eval tab runs a set and shows the results case by case, with sliders to set each metric’s threshold before a run so you can feel out sensible cut-offs interactively rather than editing JSON blind.
Its real superpower is failure triage. When a case fails, the UI lets you open the replay and inspect it step by step — the actual event stream, the tools the agent called, the transfers between sub-agents, the state at each turn — next to the expectation that did not match. Reading a raw trajectory diff in a CI log tells you that case 24 failed; the UI shows you the agent called lookup_order twice when the case expected once, and lets you decide in seconds whether that is a real regression or an expectation that needs loosening. Author and debug in the UI where the visual context lives; run the committed result headless in the CLI and pytest where automation lives. The artifacts are identical JSON, so nothing is lost moving between them.
Wiring evals into CI
An eval suite earns its keep only when it runs on every change that could break it. The mechanics are the same as any test gate: because AgentEvaluator.evaluate raises on failure, a pytest invocation in your pipeline blocks a merge the moment a case drops below threshold. What makes agent CI different is what triggers a run and how much you run.
- name: Agent evals
run: pytest tests/evals -q
env:
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}Tier the work to keep signal per dollar honest. A change to a tool docstring, an instruction, or agent wiring should trigger the relevant suite plus a small routing smoke set on the PR — fast, mostly the free deterministic metrics, no judge cost. A model or prompt change deserves the full suite with a raised num_runs for variance-sensitive cases, and it is worth running the LLM-judged metrics there because that is exactly the kind of change that shifts meaning without shifting trajectory. Push the heaviest work — the whole suite with judge scoring against the production model — to a nightly job that catches dependency and model drift before a user does. Smoke on every PR, full suite nightly: the split that keeps evals both trustworthy and affordable.
Gotchas and operating tips
A few things bite in practice. Field-name drift: the flat test file uses tool_name/tool_input; the EvalSet uses name/args under intermediate_data.tool_uses — mixing them produces a case that parses but never matches. ROUGE brittleness: response_match_score rewards shared words, so a correct answer phrased differently can score low; write references in the agent’s own voice or move to final_response_match_v2 for meaning-level grading. Judge nondeterminism: LLM-judged metrics vary run to run, so pin the judge model/version and spot-audit its scores against human labels before trusting a gate built on them.
Two more. Live vs stubbed tools is a deliberate choice: live tools catch integration drift but need credentials and can be flaky in CI; stubbed tools isolate agent logic and run credential-free, which is usually what you want on a PR. And fixtures matter — a case that forgets to set session_input.state exercises the agent as an anonymous, empty-state user, which is rarely the path you meant to test. Set the state explicitly so the case tests the scenario you think it does, not an accidental cold-start variant.
.test.json for single scenarios and a .evalset.json of many curated cases, both in the EvalSet schema, each case capturing a query, an expected tool_uses trajectory, a final_response reference, and an initial session_input.state. The scoring: two independent axes — trajectory and response — graded by named metrics, the free deterministic tool_trajectory_avg_score (default 1.0) and response_match_score (default 0.8) plus LLM-judged metrics like final_response_match_v2, with thresholds set in test_config.json’s criteria. The runners: adk eval for the CLI, AgentEvaluator.evaluate for pytest and CI, and adk web for authoring and step-by-step failure triage. Capture cases from real sessions, hold trajectory strict and let response breathe, tier PR smoke tests against a nightly full run, and a stochastic agent becomes something you can refactor and migrate without flying blind.