Core concepts

  • Dataset — inputs + expected outputs.
  • Runner — invokes the agent on each row.
  • Scorers — score outputs against expectations.
  • Report — aggregates + shows regressions.
Advertisement

Minimal setup

EvalDataset ds = EvalDataset.fromJsonl(Path.of("evals/happy_path.jsonl"));
EvalRunner runner = EvalRunner.builder()
    .agent(myAgent)
    .dataset(ds)
    .addScorer(new ExactMatchScorer())
    .addScorer(new SemanticSimilarityScorer())
    .build();
EvalReport report = runner.run();
Advertisement

Local + CI

Run locally when iterating. Run in CI on every PR to catch regressions.