Research review / researchradar

Agent Reliability Has Two Kinds of Noise, and Averages Mix Them

A November 14 arXiv paper proposes a scenario, instance and sample hierarchy for agent evaluation, with exact finite-sample variances for pass@k and pass^k. Its central point is separating run-to-run randomness from differences between tasks.

By Sandeep Belgavi · Published 2026-11-22

A Hierarchical Statistical Framework for Evaluating Agent Reliability by Mirco Milletarì and Tuhin Kundu was submitted to arXiv on November 14, 2026. It arrives the same week ADK Python began reporting pass@k and pass^k side by side, which makes its question practical rather than academic.

A hierarchy of scenarios, instances and samples

Per the abstract, the authors define three levels: scenarios (kinds of situations), instances (specific tasks within a scenario) and samples (repeated runs of one instance). They apply this to mean pass rate, pass@k and pass^k with uncertainty estimates for each, independently derive exact finite-sample variances for the unbiased pass@k and pass^k estimators, and note that related results appear in other recent submissions. The stated central contribution is a constructive derivation of how uncertainty propagates up the hierarchy, connected to coarse-graining in statistical physics, so that execution noise is separated from task heterogeneity.

The abstract motivates the work with a benchmark in which even strong models perform inconsistently across business scenarios. This digest read the abstract only; no figures from the paper body are reproduced.

Why the two kinds of variation need different fixes

Analysis: a single pass rate blends two questions. If an agent fails a task on some runs and passes it on others, the fix is usually about stability: sampling settings, retries, deterministic tool handling. If it passes some tasks every time and fails others every time, the fix is about capability or coverage: missing tools, unclear policy, poor retrieval. The same 70% aggregate can come from either pattern, and a confidence interval that ignores the hierarchy can be far too narrow when you have many runs of only a few tasks.

A worked illustration with synthetic numbers

Calculation on invented data, not results from the paper: Belgavi's Book Store evaluates a help-desk agent on two scenarios, returns and order status, with five instances each and ten runs per instance. In design A every instance succeeds on 7 of 10 runs. In design B, seven instances succeed 10 of 10 times and three succeed 0 of 10 times. Both report a 70% mean pass rate. Design A has pass^3 of C(7,3)/C(10,3) = 35/120, about 0.29, on every instance. Design B has pass^3 of 1.0 on seven instances and 0 on three, a mean of 0.70. Dinky can compute both in a spreadsheet. The averages match and the engineering responses are opposite.

What to watch

Whether evaluation frameworks start reporting variance by level, not just per-case summaries, and whether the variance results are confirmed by the other submissions the authors mention.

Sources accessed November 22, 2026. Views are my own and do not represent any employer, past or present.

Sources and evidence

Report agent reliability by level: run-to-run noise and task-to-task differences need different fixes, and one pass rate can hide which one you have.