Online Evaluation of a Production Agent: Stratified Sampling, a Judge Budget and Telling a Traffic-Mix Shift From a Real Quality Drop

The aggregate pass rate moved. Did the agent change?

Offline evaluation tells you how an agent behaves on a fixed dataset. Once the agent is in production, the question changes to whether it is still behaving well on the traffic it actually gets, and that traffic is not fixed. A typical setup samples a few hundred finished runs a week, has a judge model or a reviewer score each one against a rubric, and plots the weekly pass rate. The plot is easy to build and easy to misread.

The pass rate of a whole population is a weighted average of the pass rates of its parts, with the traffic mix as the weights. If next week brings more of the hard requests, the average falls even when nothing about the agent changed; if it brings more of the easy ones, the average rises and can hide a real regression in a small segment. The Stanford Encyclopedia of Philosophy entry on Simpson's paradox discusses the general phenomenon: an association in the aggregate can differ from, or reverse, the association within every subgroup. For an online evaluation the practical consequence is simple: never alert on the aggregate alone.

Segments, a judge budget and intervals

Start by choosing segments that plausibly differ in difficulty and that you can assign to every run without a model: the product area of the request, the tool set the run used, the language, or the customer tier. Three to eight segments is enough; more segments spread the judging budget too thin.

The judge budget is the number of runs you can afford to score in a period, whether the scorer is a person or a model. How you spend it decides how precise each segment estimate is. Uniform random sampling spends it in proportion to traffic, so small segments get few samples and wide intervals. Stratified sampling chooses a number of samples per segment up front. Equal allocation gives every segment the same precision; Neyman allocation, which samples each segment in proportion to its traffic share times the standard deviation of its outcome, minimises the variance of the overall estimate. For a pass rate the standard deviation is the square root of p(1 - p), so segments near 50 percent get more samples than segments near 95 percent.

Every per-segment rate needs an interval. With a few dozen samples the normal approximation is poor, especially near 0 or 1; the Wilson score interval from 1927 behaves well at small sizes and is a few lines of code. Evan Miller's paper on error bars for language model evaluations makes the broader case: an evaluation is an experiment, and a point estimate without a standard error is not a result.

Illustrative python, checked for syntax; adapt names and storage to your system.

from math import sqrt

def wilson(passed, n, z=1.96):
    """95% Wilson score interval for a pass rate."""
    if n == 0:
        return (0.0, 1.0)
    p = passed / n
    d = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / d
    half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return (centre - half, centre + half)

def report(samples, baseline, reference_mix):
    """samples: {segment: (passed, judged)}; baseline: {segment: rate}; reference_mix sums to 1."""
    rows, reweighted = [], 0.0
    for seg, (passed, n) in samples.items():
        rate = passed / n
        low, high = wilson(passed, n)
        rows.append((seg, n, round(rate, 3), round(low, 3), round(high, 3), high < baseline[seg]))
        reweighted += reference_mix[seg] * rate
    return rows, round(reweighted, 3)

The last column of each row is the alert: the whole interval sits below the segment's baseline rate. The reweighted figure answers the question the aggregate was supposed to answer: what the pass rate would be this week if the traffic mix were the same as in the reference period.

Worked example: the aggregate rose and one segment got worse

The traffic and rates here are synthetic. A support agent handles three segments. In the reference week the mix was 50 percent billing, 30 percent account and 20 percent technical, and the judged pass rates were 0.92, 0.85 and 0.70, so the aggregate was 0.855.

First scenario: next week the mix shifts to 30, 30 and 40 percent because a product launch brings technical questions, and the agent is unchanged. The aggregate falls to 0.811, a drop of more than four points, and an aggregate alert fires although every segment is exactly as good as before. Reweighting to the reference mix gives 0.855 again and shows the drop is all mix.

Second scenario, the dangerous one: the mix shifts towards billing, 70, 20 and 10 percent, and a tool change makes the technical segment worse, from 0.70 to 0.55. The aggregate rises to 0.869. An aggregate dashboard shows an improvement. Splitting the change from 0.855 to 0.869 into its parts gives a mix effect of plus 2.9 points and a rate effect of minus 1.5 points; reweighted to the reference mix the week scores 0.825, three points worse than the reference.

With uniform sampling the regression is detected only just, and with a slightly smaller technical share it would have been missed: 13 passes out of 24 gives an interval whose upper end, 0.721, is above the baseline. Equal allocation detects it comfortably with the same budget. The lab linked below lets you move the mix, the regression and the allocation and watch each alert rule decide.

Choosing the reference mix and the baseline

The reference mix should be a period you trust and that is long enough to be stable, such as the four weeks before the last model or prompt change. Keep it fixed until the next deliberate change, then roll it forward explicitly; a rolling reference that moves every week absorbs slow drift into the baseline and never alerts on it. Store the per-segment baselines with their own intervals, because a baseline estimated from 80 samples is itself uncertain, and a fair comparison uses a test on the difference of two proportions rather than a fixed number.

Watch the segments themselves as a separate signal. A sudden change in the mix is often the more urgent news: a new product, a broken upstream classifier sending everything to one segment, or an abuse campaign. Report the mix beside the rates every week.

The judge is part of the measurement

If a model scores the runs, its error is part of every number above. Measure its agreement with human reviewers per segment before trusting it, because a judge that is reliable on billing questions may be unreliable on technical ones, and that difference would look exactly like a quality regression. The judge calibration guide covers the agreement measurements; the point here is that they must be stratified the same way as the evaluation. Re-check agreement whenever the judge model, the rubric or the agent's output format changes.

Running it every week

1. Assign a segment to every finished run at write time, from fields the runtime already records.
2. Draw the week’s sample by stratum with a fixed seed, so the selection can be reproduced and audited.
3. Score the sample with the judge and a smaller human-reviewed subset that keeps measuring judge agreement.
4. Compute per-segment rates with Wilson intervals, the reweighted aggregate and the mix.
5. Alert per segment when the interval sits below the baseline; report mix changes separately.
6. Write each weekly report to storage with the seed, the judge version and the rubric version.

The online_eval_mix_shift example in agent-decision-lab implements these steps on synthetic run records, with tests for both scenarios above.

Failure modes

• Alerting on the aggregate, which fires on mix shifts and stays silent on regressions in small segments.
• Segments defined by a model, so a classifier change moves runs between segments and looks like a quality change.
• Sampling only runs that ended in an error or a complaint, which measures the tail and calls it the population.
• A rolling baseline that absorbs slow drift; fix the reference and move it deliberately.
• Judge drift after a judge model update, misread as agent drift; keep a human-reviewed control subset.
• Reporting rates without intervals, so a two-point move on 40 samples triggers an incident.

What this does not replace

Online evaluation tells you whether quality on live traffic has moved; it does not tell you why. Pair it with trajectory-level regression analysis, such as detecting behaviour regressions from the event stream, and with the offline suite that gates releases. It also does not measure outcomes the rubric cannot see, such as whether a customer came back a week later; join those business signals by run id when you have them.

Practice and related reading

Hands-on: Online agent evaluation: traffic mix shift versus quality drop, an interactive lab on synthetic data.

Further reading:

• Agent evaluator architecture
• LLM-as-judge scorer with calibration against humans
• Detecting agent behaviour regressions
• Human evaluation and inter-rater agreement

Sources and scope

Checked against these sources on 20 October 2026. Examples and figures are illustrative, not measurements.

• Evan Miller, Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv 2411.00640)
• Stanford Encyclopedia of Philosophy: Simpson’s Paradox
• E. B. Wilson, Probable Inference, the Law of Succession, and Statistical Inference, JASA 22 (1927)