Tool Selection at Scale: Why Hundreds of Tools Break Agents and How to Route

What a big catalog costs on every call

Tool definitions are part of the prompt. A definition is a name, a description and a JSON schema for the arguments; with a few parameters and a two-sentence description, 100 to 200 tokens is typical. Multiply by the catalog:

# illustrative: 150 tokens per tool definition, 1,500-token system prompt, $3 per million input tokens
tools in prompt     prompt tokens     input cost per 1k calls
      10                3,000                 $9
      30                6,000                $18
     300               46,500               $140
     900              136,500               $410

Those tokens are paid on every model call in an agent loop, not once per conversation, and they add prefill latency on every call too. Prompt caching can make a stable tool block cheaper to resend; check your provider's caching rules, and note that any change to the tool list (a new tool, a reordered list, an updated description) invalidates the cached prefix. A 900-tool prompt also eats a large share of the context window that the conversation itself needs.

Why accuracy falls as the list grows

Picking a tool is a classification problem whose number of classes is the size of the catalog. Two effects make it harder as the catalog grows. First, there are more near neighbours: in a 30-tool catalog there may be one other order-related tool; in a 300-tool catalog there may be ten. Second, the model must attend over a longer prompt to compare them. Public function-calling evaluations such as the Berkeley Function Calling Leaderboard include scenarios with multiple and irrelevant functions precisely because selection degrades when the candidate list contains distractors, and model providers' own guidance recommends keeping the number of functions in a request small. OpenAI's function-calling guide, for example, has suggested staying under about 20 functions at a time; check the current wording for your provider.

The lab models this mechanism explicitly. Each tool and request is a vector; tools in the same domain are close together. The model's choice is the tool with the highest similarity to the request plus random noise, and the noise grows with the logarithm of the number of tools it has to read: noise = 0.01 + 0.012 · ln(n). The constants are illustrative, but the consequence is general: with 300 tools in the prompt, a sibling tool beats the right one often enough that only about 57% of requests are routed correctly in the lab, against well over 90% with a shortlist of ten.

Retrieval: let a cheap search build the shortlist

The standard fix borrows from retrieval-augmented generation. Embed every tool's name and description once. At each step, embed the request (or the agent's current sub-goal), retrieve the k most similar tools, and put only those in the prompt. Research systems such as Gorilla and ToolLLM paired a retriever with the model for exactly this reason when working with thousands of APIs.

-- pgvector: tools(name text, description text, schema jsonb, embedding vector(768), enabled bool)
SELECT name, description, schema
FROM tools
WHERE enabled
ORDER BY embedding <=> $1          -- cosine distance to the request embedding
LIMIT 10;

def tools_for_step(request, state, k=10):
    core = CORE_TOOLS                        # always available: search_tools, ask_user, finish
    hits = vector_search(embed(rewrite_for_tools(request, state)), k=k)
    pinned = state.tools_used_recently       # keep tools the agent is in the middle of using
    return dedupe(core + pinned + hits)

Three details make the difference in production. A small set of core tools stays in every prompt, including a search_tools meta-tool the model can call when the shortlist has nothing suitable, so a retrieval miss becomes one extra step instead of a wrong action. Tools the agent is in the middle of using stay pinned across steps, so the shortlist does not change under it mid-task. And the query used for retrieval should describe the sub-goal of the current step, not the user's whole first message, which in a long task stops resembling what is needed now.

Choosing k: recall against confusion

The shortlist size trades two failure modes. If k is too small, the right tool is sometimes not retrieved, and no model can choose a tool it never sees: end-to-end accuracy is capped by the retriever's recall@k. If k is too large, the prompt grows and fills with look-alikes, and selection accuracy among the candidates falls. End-to-end accuracy is roughly the product:

P(correct) = recall@k  ×  P(model picks the right tool | it is among k candidates)

# lab, 300 tools, vague requests (calculated):
k = 3   : recall 73%  x  ~98%  ->  ~71%
k = 8   : recall 96%  x  ~91%  ->  ~87%
k = 20  : recall 98%  x  ~77%  ->  ~76%

The curve has a maximum, and where it sits depends on how well requests match descriptions. With clear requests in the lab, a shortlist of three already reaches nearly 100% recall; with vague requests, recall needs eight or more. Measure recall@k on a labelled set of real requests (with personal data removed) and pick the smallest k whose recall is close to its plateau. Mission two of the lab is this exercise: vague requests have pushed the right tool out of the top three, and the fix is a larger shortlist, but not the largest.

Grouping, hybrids and the catalog itself

Retrieval is one of several routing designs, and they combine:

In the lab the group-then-tool strategy pays for a second model call in latency and tokens, which is why it loses to a well-tuned top-k shortlist there; in systems where embeddings match poorly (very short requests, jargon) it can do better. Whatever the router, the catalog is the biggest lever. Descriptions that say what the tool is for, when not to use it, and how it differs from its neighbours help both the retriever and the model. Mission three shows this with near-duplicate tools, the "v1, v2 and v2-final" problem: the right tool is in the shortlist, so retrieval is not the problem; making descriptions distinct, or deprecating the duplicate, is.

Writing descriptions that route well

A description does two jobs at once: it is the text the retriever embeds, and it is the text the model reads when choosing among the shortlist. Most catalogs are written for neither. A useful template states the entity and action, the inputs that identify the target, when to use the tool, and when to use a neighbour instead:

# before: three tools that embed almost identically
get_order        "Gets an order."
order_lookup     "Look up order information."
orders_v2_get    "Order details (v2)."

# after: one tool kept, the others disabled or renamed for a distinct job
get_order_status "Current status, items and shipment of ONE order, by order_id (format ORD-123).
                  Use for questions about a specific existing order. For a customer's order
                  history use list_customer_orders; to change an order use update_order."

The "when not to use" sentence is what separates neighbours, and it costs a few dozen tokens per tool, which a shortlist can afford. Keep names consistent within a server (one verb convention, one noun per entity), write descriptions in the vocabulary users actually use, and add a review step to tool registration, the same way an API change gets a review. In the lab, the distinctiveness slider stands in for this work: it spreads tools further apart in embedding space, which raises both retriever recall and the model's accuracy among the candidates.

Routing is also a security boundary

The router decides which actions the model can even consider, so it belongs in the threat model. Filter by permission before retrieval: a tool the current user or agent is not allowed to call should never enter the shortlist, both because the model may try it and because its description consumes tokens for nothing. Treat tool descriptions from third-party servers as untrusted input. A description stuffed with popular keywords can win retrieval for requests it has nothing to do with, and a description that contains instructions can steer the model once it is in the prompt; the companion article on MCP tool poisoning covers that attack. Pinning tool definitions by hash, reviewing description changes, and alerting when a new tool suddenly captures a large share of routing decisions are cheap defences that also catch honest mistakes.

Knobs in real systems

The protocols and providers give you building blocks rather than a finished router. The MCP specification has servers list their tools (with pagination for large lists) and lets them notify clients when the list changes, so a client can rebuild its tool index on change instead of polling. Tool annotations describe behaviour (read-only, destructive, idempotent) and can be used to filter candidates by risk before they reach the model. On the model side, most chat APIs accept a tool list per request and a tool-choice setting that can force a specific tool or none; some providers have also introduced server-side tool search with deferred loading of definitions. Names and behaviours differ and change quickly, so check the current documentation of your provider and MCP SDK.

For the index, any vector store works at this scale: a few thousand tool embeddings fit in memory with exact search, and pgvector, Qdrant, Elasticsearch or FAISS are all reasonable if tools already live in one of them. The harder engineering is keeping the index in sync with what each MCP server currently offers and what the current user is allowed to call.

A worked example: Belgavi's Book Store tools

Suppose a hypothetical agent at Belgavi's Book Store connects four MCP servers (catalog, orders, payments and support) exposing 300 tools in total, including several near-duplicates left over from migrations. With every tool in the prompt, each call carries about 46,500 tokens, roughly $140 per thousand calls in input at the illustrative price, and the lab-style model routes only a little over half of requests correctly.

Adding a retriever that returns the top ten tools, plus three core tools including search_tools, cuts prompts to about 3,000 to 3,500 tokens and roughly $10 per thousand calls, and raises routing accuracy above 90% for clear requests. Measuring recall@k on labelled requests shows recall plateauing near k = 8, so the shortlist shrinks to eight. The near-duplicates are fixed in the catalog: deprecated versions are disabled, and the remaining descriptions say which entity each tool works on and when to prefer it. When the catalog later triples, prompt size stays the same and only recall needs rechecking. All figures are calculated from the illustrative model rather than measured.

Evaluating, monitoring and common mistakes

Evaluate the router and the model separately, or you will not know which one to fix. Build a labelled set of requests with the expected tool (or set of acceptable tools), then measure:

• Retriever recall@k for several values of k, by domain. A low value means descriptions or the retrieval query need work.
• Selection accuracy given recall: of the cases where the right tool was in the shortlist, how often the model chose it. A low value means look-alikes or unclear descriptions.
• End-to-end accuracy, tokens and latency per configuration.

In production, log the shortlist and the chosen tool on every step. Watch the rate of search_tools calls (a proxy for retrieval misses), tool-call argument validation errors (often a wrong tool with a similar schema), immediate retries with a different tool, and the size of the tool block in tokens. Re-run the labelled evaluation whenever a server adds or changes tools, because a single new look-alike can quietly take traffic from the tool it resembles.

The recurring mistakes: connecting every available MCP server "just in case"; letting teams register tools without a description review; retrieving with the user's first message instead of the current sub-goal; choosing k without measuring recall; no fallback when the shortlist misses; and leaving deprecated duplicates enabled. The lab's missions walk through the main ones. Mission one replaces a 300-tool prompt with a shortlist. Mission two finds the right k when vague requests lower recall. Mission three fixes near-duplicates in the catalog rather than in the retriever. Mission four triples the catalog under a cost and latency budget, where shortlist size and description quality have to be tuned together.

Run the lab: Tool routing at scale: find the right tool among hundreds

The lab builds a synthetic catalog of tool embeddings grouped into domains, then routes a few hundred synthetic requests. For each request it runs an exact top-k search over the catalog (or a group-then-tool choice), and a noisy model picks among the candidates in its prompt. The picture shows tools laid out by domain, one sample request (white dot), the right tool (green ring), the shortlist (grey rings) and the model's pick (arrow). Start with every tool in the prompt, then load mission one.

Prefer a full-width view? Open the standalone lab. Your stars are saved in this browser and shown on the series hub.

What to do next

1. Count the tools and definition tokens your agent sends per call, per MCP server, and record how often each tool is actually used.
2. Build a labelled set of real requests with expected tools, and measure retriever recall@k and selection accuracy separately.
3. Add a retriever with a small set of always-on core tools, including a search_tools fallback, and pin tools in active use across steps.
4. Choose k as the smallest value near the recall plateau for your requests, not by habit.
5. Review tool descriptions for distinctness: purpose, when not to use, how it differs from neighbours; disable deprecated duplicates.
6. Rebuild the tool index when an MCP server reports a changed tool list, and re-run the routing evaluation on every catalog change.

Sources and further reading

• Gorilla: Large Language Model Connected with Massive APIs
• ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024)
• Berkeley Function Calling Leaderboard
• Model Context Protocol specification: Tools
• OpenAI documentation: Function calling
• pgvector: open-source vector similarity search for Postgres

Related in this series:

• Supervisor or Swarm: When Multi-Agent Systems Actually Beat One Agent
• MCP Tool Poisoning: How One Malicious Description Hijacks an Agent
• Why Agent Loops Run Away and How Step Budgets Stop Them

Diagrams, examples and the lab model are original educational material. Lab numbers are calculated by a synthetic model, not measured on real hardware or services.

Views are my own and do not represent any employer, past or present.