ADK Java Evaluation — Test Cases, Metrics, and CI Integration

Why it matters

Without evaluation, agent quality drifts. New prompts, model updates, or tool changes silently regress behavior. Evaluation is how you measure and prevent this.

The architecture

Test cases: input (user message), expected output pattern (regex, semantic match, or exact), optionally expected tool calls.

Runner in eval mode: runs agent against each test case, records actual behavior.

How it works end to end

Metrics: exact match, semantic similarity (via LLM judge), tool call correctness (right tool with right args), latency.

Regression detection: track pass rate over time; alert on drops.

CI integration: run eval on every PR; fail if pass rate drops below threshold.

Human evaluation for open-ended output; automated for structured.