Building an Agent Eval Harness

Two artifacts: a test file and an eval set

ADK stores expected behaviour in two file kinds that share one schema but serve different jobs. A test file (*.test.json) is the lightweight unit — conventionally a small, single-session scenario you keep next to the code it exercises and run in pytest. An eval set (*.evalset.json) is the heavier integration artifact: many eval cases grouped around a capability (refunds, order lookup, escalation), typically multi-turn, and usually authored through the web UI by saving real sessions.

The practical division is scale and provenance. Test files are hand-sized and hand-authored — you write a couple by hand to pin one behaviour and check them into the same folder as the agent. Eval sets are curated collections, often dozens of cases, versioned as a suite and owned jointly by product and engineering. Both are just JSON conforming to ADK’s EvalSet schema, so the runner treats them identically; the naming convention (.test.json vs .evalset.json) signals intent to humans, not to the tool. Start with test files while a capability is small, graduate to an eval set once you have enough cases that a shared, browsable collection earns its keep.

Advertisement

What a single eval case captures

An eval case is a recorded expectation about one conversation. Four things make it up. First, the query — the user turn(s) that drive the agent, stored as user_content with a parts array and role: "user". Second, the expected tool trajectory — the ordered list of tool calls the agent should make to satisfy the query, stored under intermediate_data.tool_uses, each entry a name plus an args object. Third, the reference response — the expected final answer, stored as final_response with its own parts and role: "model". Fourth, the initial session state — the fixture the replay starts from, stored in session_input as app_name, user_id, and a state dictionary.

That fourth field is what makes cases realistic without external setup. A refund case can begin from {"authenticated": true, "tier": "gold"} so the agent is exercised as a logged-in gold-tier user, no login flow required. Multi-turn cases string several invocations together in the conversation array; each invocation carries its own query, expected trajectory, and reference, so the harness scores a whole dialogue turn by turn rather than only a final answer.

Advertisement

The EvalSet JSON, up close

Here is a trimmed but faithful eval set with one case. The nesting is the thing to internalize: eval_cases → a conversation of invocations → each invocation’s user_content, final_response, and intermediate_data.tool_uses, with session_input hanging off the case as its fixture.

{
  "eval_set_id": "refund_flow",
  "name": "Refund capability",
  "description": "Broken-item and mistaken-order refund paths.",
  "eval_cases": [
    {
      "eval_id": "broken_item_refund",
      "conversation": [
        {
          "invocation_id": "inv-1",
          "user_content": {
            "role": "user",
            "parts": [{"text": "My blender arrived broken, order A-4471."}]
          },
          "intermediate_data": {
            "tool_uses": [
              {"name": "lookup_order", "args": {"order_id": "A-4471"}},
              {"name": "issue_refund", "args": {"order_id": "A-4471"}}
            ],
            "intermediate_responses": []
          },
          "final_response": {
            "role": "model",
            "parts": [{"text": "I've verified order A-4471 and issued a refund."}]
          }
        }
      ],
      "session_input": {
        "app_name": "customer_support",
        "user_id": "user-9",
        "state": {"authenticated": true, "tier": "gold"}
      }
    }
  ]
}

You rarely type this by hand end-to-end — the web UI emits it from a recorded session — but you will hand-edit it to tighten expectations, so knowing which field is which is not optional.

The lightweight test-file shape

Because the full EvalSet schema is verbose, ADK has long supported a flatter, human-friendly test-file form for the common single-turn case: a JSON array where each entry pairs a query with an expected_tool_use list and a reference answer. It is the fastest way to pin a simple tool-calling behaviour without the invocation scaffolding.

[
  {
    "query": "What's the weather in Paris?",
    "expected_tool_use": [
      {"tool_name": "get_weather", "tool_input": {"city": "Paris"}}
    ],
    "reference": "It's currently clear and about 18 degrees in Paris."
  }
]

Note the field names differ from the EvalSet form — here a tool call is tool_name/tool_input, whereas inside intermediate_data.tool_uses it is name/args. Do not cross them; that mismatch is the classic sign of a hand-copied file that will silently fail to parse the trajectory. New work trends toward the EvalSet schema (it is what the UI produces and what multi-turn cases need), but the flat form remains a pragmatic choice for a quick, single-turn regression check.

Two dimensions: did it act right, did it answer right

The harness scores every replay on two independent axes, and keeping them separate is the central design idea. The first is tool-trajectory match: did the agent call the expected tools, with the expected arguments, in the expected order? The second is response quality/match: does the final answer say what the reference says?

They are separate because either can pass while the other fails, and each failure means something different. An agent that produces the right words with the wrong trajectory — answering a refund question without ever calling lookup_order — got lucky and will hallucinate on the next input; only trajectory scoring catches it. An agent with a perfect trajectory but a garbled final message did the work and then fumbled the summary; only response scoring catches that. Endpoint-only testing collapses these into one and misses the first entirely, which is precisely why agent evaluation needs a purpose-built harness rather than a string comparison. Every built-in metric slots into one of these two axes, and a case’s verdict is the conjunction: it passes only when both the trajectory and the response clear their thresholds.

The built-in metrics

ADK ships a set of named metrics you compose per suite. The two workhorses are cheap and deterministic; the rest bring an LLM judge for what similarity cannot measure.

Metric key	Axis	What it measures	Default
`tool_trajectory_avg_score`	Trajectory	Exact match of the tool-call sequence (names + args + order), averaged over steps	1.0
`response_match_score`	Response	ROUGE-1 similarity of the final answer to the reference	0.8
`final_response_match_v2`	Response	LLM-judged semantic equivalence to the reference	rubric floor
`rubric_based_final_response_quality_v1`	Response	LLM-judged quality against a rubric, no reference needed	rubric floor
`hallucinations_v1`	Response	Groundedness of the answer in tool results / context	rubric floor
`safety_v1`	Response	Harmlessness / policy compliance of the response	rubric floor

Reach for tool_trajectory_avg_score and response_match_score first: they are free, fast, and calibration-free, so they belong on every smoke-test PR. Because response_match_score is ROUGE-based it rewards lexical overlap, so keep references phrased the way the agent actually answers or the score punishes harmless wording changes. When you need to grade meaning, open-ended quality, groundedness, or safety — things ROUGE is blind to — add the LLM-judged _v2/_v1 metrics, and pin the judge model/version so scores are comparable across runs.

Thresholds and test_config.json

A metric produces a number; a threshold turns it into a verdict. Thresholds live in a test_config.json placed beside your eval files, under a criteria object keyed by metric name. Any metric you list is active; anything you omit is not scored. When no config is present, ADK applies the built-in defaults — 1.0 for tool_trajectory_avg_score and 0.8 for response_match_score.

{
  "criteria": {
    "tool_trajectory_avg_score": 1.0,
    "response_match_score": 0.7
  }
}

The tuning instinct that keeps a suite useful: hold trajectory strict and let response breathe. A perfect 1.0 on trajectory is reasonable to demand — the agent either called the right tools in the right order or it did not, and slippage there is a real regression. Response match is where phrasing variance lives, so a threshold in the 0.6–0.8 band usually catches wholesale answer regressions without flagging every reworded sentence. Set it too high and the suite cries wolf on cosmetic diffs until people stop reading it; too low and a genuinely wrong answer sails through. Tune against a handful of known-good and known-bad runs, not by guessing.

Authoring cases: capture, then curate

The cheapest good eval case is a real conversation you liked. In adk web, drive the agent through a scenario — a clean refund, an instructively broken one — then open the Eval tab and save that session into an eval set. ADK records the actual queries, the tool calls the agent made, and the responses it produced, writing them into the EvalSet schema for you. You then curate: trim the trajectory to the calls that matter, fix up an args value, replace a mediocre final answer with the reference you actually want, and set the initial state.

Curation-from-reality beats invention on every axis. The captured trajectory is already syntactically valid and uses real tool names and argument shapes, so you are editing a working artifact rather than guessing field names into a blank file. It also captures behaviours you would not think to write — the exact argument the model tends to pass, the intermediate step it takes — which is where regressions actually hide. Treat saved sessions as raw material: capture generously, then keep only the expectations you are willing to defend as a specification, because every field you leave in becomes a thing future changes must satisfy.

Running evals: the CLI

The command-line runner is the fastest path from a file to a verdict, and the natural fit for scripts and CI. Point adk eval at your agent module and an eval file:

# run an entire eval set
adk eval path/to/agent path/to/refund.evalset.json

# supply thresholds and print per-case detail
adk eval path/to/agent path/to/refund.evalset.json \
  --config_file_path=path/to/test_config.json \
  --print_detailed_results

# run only specific cases by eval_id
adk eval path/to/agent path/to/refund.evalset.json:broken_item_refund,mistaken_order

The first positional argument is the agent module path; the second is the eval file. --config_file_path supplies the test_config.json criteria (otherwise defaults apply), and --print_detailed_results expands the output from a pass/fail summary to per-case, per-metric scores — indispensable when triaging a failure, because it shows you which axis dipped and by how much. Appending :id1,id2 to the file path runs a subset, which is how you iterate on one stubborn case without paying for the whole suite each loop.

Running evals: pytest

For CI and for keeping evals next to your other tests, ADK exposes AgentEvaluator.evaluate, an async helper you call from a normal pytest test. It runs the eval file, applies the thresholds, and raises an assertion failure if any case falls short — so a failing eval is just a failing test, with no bespoke reporting to wire up.

import pytest
from google.adk.evaluation.agent_evaluator import AgentEvaluator

@pytest.mark.asyncio
async def test_refund_flow():
    await AgentEvaluator.evaluate(
        agent_module="customer_support.agent",
        eval_dataset_file_path_or_dir="tests/evals/refund.evalset.json",
        num_runs=2,
    )

agent_module is the import path to the module that exposes your root agent. eval_dataset_file_path_or_dir accepts a single file or a directory — pass a folder and it runs every eval file inside, which is how you gate a whole capability with one test. num_runs replays each case N times to sample variance: agents are stochastic, so a case that passes once may be a coin flip, and averaging over a few runs turns a flaky gate into a meaningful one. A sibling test_config.json in the same directory supplies the thresholds automatically.