ExactMatchScorer

Pass if output == expected. Use for tool_name selection, classification labels, structured extraction.

Advertisement

SemanticSimilarityScorer

Embeds output + expected, scores by cosine similarity. Use for open-ended text where phrasing varies.

Advertisement

Threshold tuning

EvalRunner.builder()
    .addScorer(new SemanticSimilarityScorer(0.85))  // pass if > 0.85
    .build();