Evidence label: deterministic synthetic model. This page runs a small JavaScript model of a tool-calling agent. Each synthetic task needs a random number of successful steps; on every step the agent may make progress, hit a tool error and retry, or fall into a loop of identical calls. A step budget, an optional repeat detector and a choice between stopping and nudging decide how runs end, and every step is billed on the full context it re-reads. Nothing here calls a model or a tool; all numbers are calculated estimates that make the mechanism visible.
Play the lab
How to play
Move the step budget, choose a loop detector threshold and what happens when it fires, and set how many tokens of each tool result the agent keeps. Every change re-runs the same 1,500 tasks with the same random seed, so differences come from your change. The tiles show success rate, cost per thousand tasks, p95 steps, the share of tasks that hit the budget, wasted steps and p95 wall time, with green or red deltas against your previous setup. Use New random seed to check that a conclusion survives different tasks.
Missions unlock in order. Load mission setup starts from a broken configuration; fix it, then try for three stars by meeting the target more cheaply. The three event buttons stress your setup with a loop-prone model, a flaky tool and long tasks.
What the model assumes
Tasks need a lognormally distributed number of progress steps around the chosen mean (2.5 times more for 15% of tasks with the long-tail event). On each step the agent enters a repeat loop with probability 2.5% (7.5% with the model-update event), otherwise a tool error occurs with probability 6% (30% with the flaky tool), otherwise it makes progress. Keeping under 600 tokens of tool output adds errors because the needed field is cut off. A looping agent escapes on its own with probability 8% per step; a nudge escapes with probability 75%. Each step costs input tokens for a 2,500-token prefix plus all earlier tool output and assistant messages, at illustrative prices of $3 and $15 per million input and output tokens.
What it cannot tell you
Real loops are not memoryless, real tool errors cluster in time, and real models sometimes produce wrong answers rather than failing visibly, which this model does not represent: a task here either completes or does not. It ignores prompt caching, parallel tool calls, oscillation between different calls and nested agents. Use it to understand why cost grows with the square of run length and why early loop detection beats a high cap, then size your own limits from your traces.
Read the full explanation
- Why Agent Loops Run Away and How Step Budgets Stop Them (companion article)
- Agent Context Compaction: What to Keep When the Window Fills Up
- Retrying Agent Tool Calls Without Executing the Same Action Twice
All AI in Production labs · Lab catalog
- ReAct: Synergizing Reasoning and Acting in Language Models (ICLR 2023)
- Reflexion: Language Agents with Verbal Reinforcement Learning (NeurIPS 2023)
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
- LangGraph documentation: low-level concepts, recursion limit
- OpenAI Agents SDK documentation: running agents and max_turns
- Anthropic: Building effective agents
Views are my own and do not represent any employer, past or present.