Evidence label: deterministic synthetic model. The lab generates synthetic multi-step tasks with different lengths and difficulty, then runs a simulated agent on each task several times. Each step can go wrong; the agent may recover at the cost of extra steps, fail outright, or loop until it exhausts its step budget. Some tasks offer a policy-breaking shortcut at the final action. The lab computes pass@k and pass^k with the standard unbiased estimators, a confidence interval across tasks, and cost per passing run. Nothing here runs a model or a real agent.
Play the lab
How to play
Adjust the agent's sampling temperature, its step budget and whether it verifies before its final action. Then choose how you evaluate it: the grader, the k in the metric, trials per task and the number of tasks. Every change re-runs the same synthetic tasks with the same seed. Watch pass@k against pass^k in the tiles and on the chart; the gap between them is the inconsistency users feel. Inject policy-bait tasks, a flaky tool or a model update, and use the presets to compare a leaderboard-style report with a release gate. Missions unlock in order; stars reward stronger reliability, smaller evals and tighter budgets.
What the model assumes
Tasks have 3 to 9 steps and a per-step error rate drawn from a skewed distribution, so a few tasks are very hard. Each step's error rate is the task's rate plus 0.07 times the temperature. After an error, recovery probability rises from 15% at temperature zero to 80% at 0.5 and falls again above 0.6; recovery costs two extra steps. An unrecovered error either fails at once or, more often at low temperature, loops until the budget runs out. Shortcuts happen with probability 0.15 plus 0.25 times temperature; verification adds a step and catches 85% of them. The flaky tool fails 8% of steps and a retry works 90% of the time. A step costs an illustrative $0.004.
What it cannot tell you
The link between temperature and recovery is an invented assumption chosen to make an interior optimum visible; real agents may behave differently, and you should measure yours. Trials here are independent, while real failures correlate through shared tools, data and prompts. The model ignores judge errors, partial credit, user simulators and contamination. Use it to build intuition about the metrics and sample sizes, not to predict your agent's scores.
Read the full explanation
- Evaluating Agent Trajectories: Why pass@k Hides Unreliable Agents (companion article)
- Human Approval Gates That Keep Agents Safe Without Becoming the Bottleneck
- Why Agent Loops Run Away and How Step Budgets Stop Them
All AI in Production labs · Lab catalog
- Chen et al., Evaluating Large Language Models Trained on Code (pass@k estimator)
- Yao et al., tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
- Miller, Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
- Kapoor et al., AI Agents That Matter
- LangChain agentevals: trajectory evaluators
Views are my own and do not represent any employer, past or present.