Online agent evaluation: traffic mix shift versus quality drop

What this experiment demonstrates

A support agent serves three segments of traffic: billing, account and technical requests. In a reference week the mix was 50, 30 and 20 percent and the judged pass rates were 92, 85 and 70 percent, so the aggregate pass rate was 85.5 percent. This week you choose the mix and the true pass rate of each segment, and a fixed budget of judged runs is spent either in proportion to traffic, equally per segment, or by Neyman allocation. The page then computes what a weekly report would show: per-segment estimates with 95 percent Wilson intervals, the aggregate, the aggregate reweighted to the reference mix, and the split of the aggregate change into a mix effect and a rate effect.

The fixture is deterministic. Each segment's judged sample passes in exactly the expected proportion, rounded to whole runs, so the intervals show the uncertainty a real sample of that size would carry without random noise changing the result between runs. Nothing calls a model.

Run controlled comparisons

Start with the defaults: the mix has moved towards billing and the technical segment has dropped from 70 to 55 percent. The aggregate estimate rises to 86.8 percent (the true aggregate is 86.9), so the aggregate alert is silent, while the reweighted figure falls to 82.5 percent and the technical segment alert fires, only just, because uniform allocation gave it 60 judged runs. Switch the allocation to equal per segment and the technical interval narrows to well below the baseline. Lower the technical share to 4 percent with uniform allocation and the segment alert misses the regression; the summary reports it as a missed regression.

Now set the technical rate back to 70 and move the mix to 30, 30 and 40 percent. Every segment is unchanged, but the aggregate falls by more than four points and the aggregate alert fires: a false alarm caused entirely by mix. The reweighted figure returns to about 85.5 percent, 85.6 in the rounded sample, and no segment alert fires. Shrink the budget to see intervals widen until even a large regression goes unnoticed, and raise the aggregate tolerance to see how a looser threshold trades false alarms for misses.

Inspect the trace and keep evidence

The segment table lists each segment's traffic share, judged runs, passes, estimate, Wilson interval, baseline and alert decision, with alerted rows highlighted. Metrics show the aggregate, the reweighted aggregate, the mix and rate effects in points, the number of segment alerts, whether the aggregate alert fired, and two error counts: regressions that no alert caught and aggregate alarms with no real change. Start trace, Next step, Show all and the slider control how much of the decision trace is visible; the metrics always describe the whole run. Reset restores the defaults and Export JSON downloads the inputs, the table and the trace.

Limits and production translation

Three segments with known true rates are an idealisation; real segments are fuzzy, the true rates are what you are trying to estimate, and the sample is random, so a real report can miss or flag a change by chance in either direction. The baseline is treated as exact, while a real one carries its own interval and deserves a two-sample comparison. The judge is assumed to be perfect; a real judge has errors that can differ by segment. What carries over: segment the traffic, spend the judge budget by stratum, put an interval on every rate, reweight to a fixed reference mix, alert per segment, and report the mix as its own signal.