Why architecture matters here

Evaluator fails when judges drift, rubrics are ambiguous, or trace scoring misses step failures. Architecture matters because rubric + judges + calibration determine trust.

Advertisement

The architecture: every piece explained

The top strip is the machinery. Task suite. Rubric weighted criteria. Judge model. Trace scoring step-level.

The middle row is control. Regression gate. Human calibration. Multi-judge reduce variance. Diagnostics failure taxonomy.

The lower rows are ops. Cost + latency. Feedback loop. Ops — governance + versioning + audit.

Agent evaluator — rubric + judge + trace scoring + regression + calibrationmeasure whether the agent actually helpsTask suiteproduct-representativeRubricweighted criteriaJudge modelscore outputsTrace scoringstep-levelRegression gateCI blocksHuman calibrationalign judgeMulti-judgereduce varianceDiagnosticsfailure taxonomyCost + latencyeval budgetFeedback loopuser labelsOps — governance + versioning + auditgatealignaveragediagnosebudgetlooploopoperateoperate
Agent evaluator with rubric + judges + trace scoring.
Advertisement

End-to-end flow

End-to-end: PR changes agent. Eval runs 300 tasks. Judge model scores per rubric. Trace scoring flags 2 step-level failures. Regression gate detects 1% quality drop; PR blocked. Human calibration weekly aligns judge.