Evidence label: deterministic synthetic model. The lab simulates five eight-hour shifts of tool calls from a hypothetical bookstore support agent. Each call is drawn from a fixed tool mix, may be harmful with a per-tool probability, and gets a synthetic risk score. A gate policy routes calls to a pool of reviewers who work a first-in, first-out queue, and each reviewer's chance of spotting a harmful call falls as their decisions in the last hour pile up. Every number is generated by this synthetic model; nothing here observes real reviewers, agents or attacks.
Play the lab
How to play
Choose a gate policy, then adjust the score threshold, the number of reviewers, the agent's call rate, the approval timeout and what happens when it expires. Each change re-runs the same five shifts with the same seed, so differences come from your change. Watch the six tiles: harm executed per 1,000 calls, the share of harmful calls stopped, the share of calls sent to a human, p95 approval wait, peak decisions per reviewer-hour and the share of legitimate calls that completed. Inject a prompt-injection campaign, a refund surge or a reviewer absence to stress your setup, and use the presets as starting points. Missions unlock one at a time; stars reward meeting the goal with fewer reviewers or a narrower gate.
What the model assumes
Calls arrive as a Poisson process. Tools are reads (45%), notes (20%), price changes (8%), refunds (10%), customer emails (12%) and deletes (5%); harm rates are 0 to 4% per tool. Benign calls get scores around a per-tool base and harmful calls around 72, each with noise; injected calls are harmful but score around 30. Review takes 0.4 minutes plus an exponential with mean 0.8 minutes. Detection is 97% up to 30 decisions in the previous hour and falls linearly to 47% at 90. Reviewers deny 1.5% of legitimate calls. Under the tiered policy, 70% of harmful reversible calls that skip the gate are undone by an audit. Timeouts either approve or deny the call.
What it cannot tell you
The fatigue curve, the score distributions and the undo rate are invented to make the trade-off visible; real values depend on your reviewers, tooling, card design and attackers, and must be measured with canaries and incident reviews. The model ignores reviewer skill differences, priority queues, batching of similar approvals, partial approvals, collusion and adaptive attackers who probe your threshold. Use it to reason about the shape of the problem, not to staff a team.
Read the full explanation
- Human Approval Gates That Keep Agents Safe Without Becoming the Bottleneck (companion article)
- Evaluating Agent Trajectories: Why pass@k Hides Unreliable Agents
- MCP Tool Poisoning: How One Malicious Description Hijacks an Agent
All AI in Production labs · Lab catalog
- Model Context Protocol specification: Tools (2025-06-18)
- LangGraph documentation: Human-in-the-loop
- OWASP Top 10 for LLM Applications 2025: LLM06 Excessive Agency
- Greshake et al., Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
- Debenedetti et al., AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
- Parasuraman and Manzey, Complacency and Bias in Human Use of Automation: An Attentional Integration (Human Factors, 2010)
Views are my own and do not represent any employer, past or present.