Evidence label: deterministic synthetic model. This page runs a small JavaScript model of three agent architectures. Each synthetic task touches a random set of domains, each needing two to four tool steps. A single agent chooses among every tool; a supervisor routes each subtask to a specialist holding one domain's tools; a swarm of peers hands the conversation from agent to agent. Tool-choice accuracy, routing accuracy, handoff losses and model-call costs are illustrative constants; nothing here calls a model, and all numbers are calculated estimates.
Play the lab
How to play
Choose an architecture, set the catalog size and how many domains a task touches, and for the supervisor choose the handoff payload and whether subtasks run in parallel. Every change replays the same 600 tasks with the same seed. Tiles show task success, cost per thousand tasks, p95 latency, model calls and tokens per task, with green or red deltas against your previous setup. New random seed checks whether a conclusion survives different tasks.
Missions unlock in order, each loading a broken setup. Events double the tool catalog, make every task touch two more domains, and make specialist descriptions overlap so routing gets worse.
What the model assumes
Tool definitions cost 120 tokens each and system prompts 2,500 tokens (3,500 for the supervisor). Tool-choice accuracy follows 0.995 − 0.08 · ln(tools / 16), minus a small long-context penalty; a wrong choice costs a step and ends the task 25% of the time. The supervisor routes correctly 97% of the time (88% with overlap) and retries up to twice; each specialist spends one call reading its handoff; a summary handoff loses a needed detail 2% of the time. Peers route correctly 92% of the time (72% with overlap), and a wrong peer attempts the work itself 35% of the time. Parallel subtasks that depend on an earlier one fail 25% of the time. Calls take 1.2 seconds plus 0.02 seconds per thousand input tokens; tools take 0.5 seconds; prices are an illustrative $3 and $15 per million input and output tokens.
What it cannot tell you
Real routing and tool-choice accuracy depend on your descriptions, model and domain, and errors are correlated rather than independent. The model ignores prompt caching, model tiers (a cheaper model for specialists), user interaction mid-task, and verification steps. Use it to see how catalog size, task breadth and handoff design move the trade-off, then compare architectures on your own evaluation set.
Read the full explanation
- Supervisor or Swarm: When Multi-Agent Systems Actually Beat One Agent (companion article)
- Tool Selection at Scale: Why Hundreds of Tools Break Agents and How to Route
- Why Agent Loops Run Away and How Step Budgets Stop Them
All AI in Production labs · Lab catalog
- Anthropic: How we built our multi-agent research system
- Why Do Multi-Agent LLM Systems Fail? (MAST taxonomy)
- AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation
- LangGraph documentation: multi-agent systems
- OpenAI Agents SDK documentation: handoffs
Views are my own and do not represent any employer, past or present.