Judge prompt

Rubric-driven. 'On a scale of 1-5, how well does this response answer the question? Return JSON: {score: N, reason: ...}'

Advertisement

Wire as a scorer

public class JudgeScorer implements Scorer {
  public Score score(EvalCase c, AgentResponse r) {
    var judge = judgeLLM.generate(rubric + c.input() + r.text());
    return Score.of(parseScore(judge), parseReason(judge));
  }
}
Advertisement

Cheap judge model

Gemini Flash-Lite or similar. 100x cheaper than the primary agent, quality is sufficient for judging.