Agent Human Evaluation

Why it matters

Human eval is gold standard. Expensive; blend with LLM judge.

The architecture

Rubric + guidelines.

Multiple raters.

Inter-rater agreement.

How it works end to end

Scale AI, Prolific.

Multiple raters.

Cohen's kappa.