Two Sources of Eval Noise: Separating Generation Variance From Grader Variance With Repeated Trials and Re-Grading

One number, three sources of noise

The support team at Belgavi's Book Store evaluates its returns agent on 200 synthetic help-desk cases, graded by a model judge against a rubric. Last week's score was 71 percent; this week's, after a prompt change, is 68. Is the change worse, or is that noise? Answering needs to know how noisy the score is, and that noise has three sources that behave differently.

• Task sampling. The 200 cases are a sample of the cases the agent will meet. A different 200 would give a different score even with a perfect, deterministic agent and grader.
• Generation. At a non-zero temperature, and often even at zero, the agent produces a different trajectory each time it runs a case.
• Grading. A model judge reading the same transcript twice can give two verdicts. So can two human raters.

Evan Miller's paper on error bars for language model evals treats the first two: it splits score variance, using the law of total variance, into the variance between questions and the expected variance within a question, and shows that resampling answers reduces only the second. Grader noise adds a third level with the same structure. Each generation can be graded more than once, and grader disagreement on a fixed output is a variance you can measure directly.

The arithmetic

Write a score as the task's true mean plus a generation deviation plus a grading deviation. With n tasks, K generations per task and G grades per generation, all averaged, the variance of the eval mean is:

Variance of an eval mean with nested repeats; assumes independent generations and independent grades.

var(mean) = (task_var + generation_var / K + grader_var / (K * G)) / n

Task variance shrinks only with more tasks. Generation variance shrinks with more tasks or more generations per task. Grader variance shrinks with any of the three. Plugging in illustrative components, task 0.10, generation 0.08 and grader 0.03, for a pass/fail score gives these 95 percent half-widths:

With these components, the drop from 71 to 68 is well inside the noise of a single run, and no amount of re-running or re-grading the same 200 cases gets the half-width below 4.4 points: the floor is set by task variance. Re-grading alone barely helps here, because grader variance is the smallest component. With a noisier judge the ranking changes, which is why it is worth measuring rather than assuming.

Estimating the components

You need a design with repeats at both levels on at least a subset of tasks: K of at least 2 generations per task and G of at least 2 grades per generation. Then three averages give the three components, by the method of moments.

Illustrative python, checked for syntax; adapt names and storage to your system.

import random
from statistics import mean, variance

def components(scores):
    """scores[task][sample][grade] -> (task, generation, grader) variance estimates."""
    K, G = len(scores[0]), len(scores[0][0])
    grader = mean(variance(grades) for task in scores for grades in task)
    per_output = [[mean(grades) for grades in task] for task in scores]
    generation = mean(variance(outs) for outs in per_output) - grader / G
    task_means = [mean(outs) for outs in per_output]
    task = variance(task_means) - generation / K - grader / (K * G)
    return task, generation, grader

def synthetic(n=300, K=4, G=3, flip=0.1, seed=7):
    rng = random.Random(seed)
    data = []
    for _ in range(n):
        p = rng.choice([0.05, 0.5, 0.95])           # hard, coin-flip and easy tasks
        task = []
        for _ in range(K):
            correct = rng.random() < p              # one generation
            task.append([int(correct != (rng.random() < flip)) for _ in range(G)])
        data.append(task)
    return data

print([round(v, 3) for v in components(synthetic())])
# [0.08, 0.075, 0.094]

The synthetic grader flips each verdict with probability 0.1, so its true variance per output is 0.1 times 0.9, or 0.09; the estimate is 0.094. The generation estimate, 0.075, sits next to the 0.074 those settings imply, and the task estimate, 0.080, next to 0.086; with 300 tasks the task term is the least precise of the three. Note what the flips do to the other two: a judge that flips binary verdicts with probability f shrinks every real difference by a factor of 1 minus 2f, so task and generation variances are scaled by its square, 0.64 here. A noisy judge does not only add noise. It also compresses the signal you are trying to measure, including the difference between two agent versions. The companion example in agent-decision-lab checks the estimator against these known values.

Where the next sample should go

Once you have the components, compare what an extra unit of budget buys. If a generation costs an agent run with tool calls and a grade costs one judge call, adding grades is cheap but only reduces the grader term. A practical procedure:

1. Run a pilot on 30 to 50 tasks with K = 3 and G = 3 and estimate the three components.
2. If grader variance is large relative to generation variance, fix the grader first: tighten the rubric, ask for evidence before the verdict, or replace judgement with a programmatic check where the task allows it. Re-grading is a fallback.
3. If generation variance dominates, run each task more than once and report the per-task mean, as the paper recommends for resampling.
4. If task variance dominates, add tasks. Nothing else helps, and it is the usual case for broad evals.
5. When comparing two agent versions, run both on the same tasks and analyse paired differences per task; shared task difficulty cancels, which the paper also recommends.

Reporting

• Report the score with its standard error and the design: tasks, generations per task, grades per generation.
• Report the grader agreement on repeated grades of the same outputs, so readers can see how much of the noise is the ruler.
• When tasks come in groups, such as several cases built from one customer story, use clustered standard errors, as the paper describes; ignoring clusters understates the error.
• Version the judge prompt and model with the result; a judge change is a change of ruler.

Failure modes

• Cached generations counted as repeats. If a response cache returns the same output for every repeat, generation variance looks like zero.
• Correlated grades. Re-grading with the same judge, prompt and seed may return the same verdict for reasons unrelated to the output. Vary the seed or sample at the same temperature you use in production grading, and treat agreement as an upper bound on reliability.
• Negative estimates. Method-of-moments estimates can go below zero on small pilots. Clip at zero and collect more data rather than reading meaning into them.
• Pass/fail hiding partial credit. Binary scores have the most variance per item; rubric scores on a scale, where defensible, usually carry more information per grade.

Practice and related reading

Further reading:

• Agent evaluator architecture: grading contracts and pass^k
• LLM-as-judge calibration and bias
• Evaluating agent trajectories

Sources and scope

Checked against these sources on 1 November 2026. Variance components in the half-width table are illustrative. The estimator output was produced by running the printed code locally with Python 3.13; no model or judge was called.

• Evan Miller: Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv:2411.00640)
• Anthropic: Building effective agents (stopping conditions, guardrails)