Why architecture matters here
The architecture matters because context is a scarce, expensive, and accuracy-sensitive resource, and tool definitions are among the greediest consumers of it. Every tool you bind costs tokens for its name, description, and parameter schema — often hundreds of tokens each — and those tokens are paid on every single model call in the turn. Bind a thousand tools and you may spend more of your context budget describing tools than doing the task, if it fits at all. Retrieval keeps that cost flat: whether the catalog has fifty tools or fifty thousand, the model sees a dozen, and the prompt size is constant.
It matters because model accuracy degrades as the option set grows, and not gently. Presented with hundreds of similar-sounding tools, a model confuses get_user with get_user_profile with fetch_account, picks the wrong one, hallucinates parameters, or freezes. Narrowing the menu to a handful of genuinely relevant tools is one of the highest-impact accuracy improvements available in agent design — the model does its best reasoning when the choice is between a few good options, not a thousand mediocre matches.
It matters because it decouples the catalog's growth from the agent's prompt. New teams can register new tools, MCP servers can export hundreds of functions, and integrations can multiply, all without touching the agent's prompt or degrading its performance — because none of that is bound until it is retrieved as relevant. This is what lets an agent platform scale to an organization's entire API surface without every agent paying the cost of every tool. The catalog and the prompt are no longer coupled.
It matters because it turns the action surface into something bounded and governable per turn. When only the retrieved-and-filtered tools are callable, you have a natural place to enforce permissions (filter out tools the caller may not use), cost controls (rank down expensive tools), and safety (require the sensitive tool to be explicitly retrievable). The set of things an agent could do on any given turn is small, known, and logged — a far better security posture than 'the model has a thousand tools and we hope it picks wisely.'
Finally it matters because it introduces a new, silent failure mode that you must design against: the retriever is now a single point of capability loss. If retrieval has poor recall and misses the right tool, the model never sees it and the task fails with no error — the agent simply acts as though the capability does not exist. Understanding the architecture means understanding that tool-retrieval quality is now a first-class reliability concern, on par with the model's reasoning, and must be measured and defended accordingly.
The architecture: every piece explained
The foundation is the tool index: a searchable representation of every tool the agent could possibly use. For each tool you embed a text representation — its name, a natural-language description of what it does and when to use it, and often example queries or parameter summaries — into a vector, and store those vectors in a similarity index. The quality of this index is dominated by the quality of the descriptions: a tool whose description clearly states its purpose and its triggers will be retrieved when relevant, while a terse or jargon-laden description will be silently missed. The index is built once and updated as tools are added, changed, or removed.
The tool retriever is the query-time engine. It takes the current request — and usually some conversation context or a distilled statement of the immediate sub-goal — embeds it with the same model used for the index, and performs a nearest-neighbor search to pull the top-k most semantically similar tools. Semantic search is what lets 'I need to move money to my landlord' retrieve a transfer_funds tool whose description never uses the words 'landlord' or 'move'; the embedding captures meaning, not keywords. The retriever's recall — does it surface the right tool when one exists — is the metric that matters most.
The filter stage applies hard constraints that embeddings cannot express. The caller's permissions remove tools they are not authorized to use, so the model is never even tempted by a forbidden action. Availability and health checks drop tools whose backend is down. Context filters can exclude tools irrelevant to the current agent's role. This stage is where governance lives: it guarantees that no matter what the retriever surfaces, only permitted, available tools reach the model. Filtering after retrieval keeps the semantic search simple and the policy enforcement explicit.
The ranking stage orders and trims the filtered candidates into the final set. Beyond raw similarity it can weigh cost (prefer a cheap tool over an expensive one when both fit), recency (a tool just used successfully may be relevant again, or conversely should be de-emphasized to avoid loops), specificity (prefer the precise tool over a generic one), and success history. The output is a small, ordered list — often five to fifteen tools — that becomes the model's menu for this turn. Ranking is where you tune the trade-off between showing enough to cover the task and few enough to keep the model sharp.
The final component is the dynamic binding back into the model call. The selected tools' full definitions — names, descriptions, parameter schemas — are injected into the prompt or the tool-calling API for this turn only. The model reasons and emits a tool call chosen from exactly that set. On the next turn, with a new sub-goal, the whole pipeline runs again and a different set may be bound. This per-turn re-selection is what keeps the bound set matched to the evolving task, and it is why some designs also keep a small set of always-on 'core' tools bound regardless of retrieval, so the agent never loses fundamental capabilities like asking the user a clarifying question.
End-to-end flow
Trace a turn through the pipeline. A user tells a support agent, 'My last order hasn't arrived and I want a refund.' The agent framework takes this request, embeds it, and hands it to the tool retriever. The retriever searches the tool index of several hundred tools and returns the top candidates by semantic similarity: lookup_order, track_shipment, issue_refund, get_customer, and a few near-misses like cancel_subscription.
The filter stage checks the caller's context. This is a support agent operating on behalf of a verified customer, so order and shipment tools are permitted; but issue_refund requires a supervisor scope this session lacks, so it is filtered out and replaced in the candidate set — the model will not see it and cannot call it. The ranking stage orders the survivors by relevance and drops the weakest, leaving a focused set of perhaps five tools bound for this turn, plus the always-on core tools like ask_clarifying_question and escalate_to_human.
The model now reasons over this small, clean menu. It recognizes it must first find the order, so it calls lookup_order with the customer's ID. The tool executes and returns the order details, including that it shipped but shows no delivery scan. That result feeds back into the agent's context, and the loop turns again: with the new sub-goal 'find out where the shipment is,' the pipeline re-runs and this time ranks track_shipment at the top, binding a fresh set tuned to the shipment question.
The model calls track_shipment, learns the package is lost in transit, and concludes a refund is warranted. But issue_refund was filtered out for lack of scope, so the model — seeing only the tools it was given, which include escalate_to_human — escalates to a supervisor with a summary rather than silently failing. The filtering did not just prevent an unauthorized action; it steered the agent into the correct alternative path, because the bound set encodes what the agent is allowed to do right now.
Now consider the counterfactual that reveals the architecture's sharpest edge. Suppose the retriever's index had a poorly written description for track_shipment — just 'shipment ops' — so it scored low and did not make the top-k. The model, never shown that tool, cannot call it; it flails, perhaps asking the user for tracking information it should have fetched itself, and the experience degrades with no error logged anywhere. The turn 'succeeded' from the model's view — it used the tools it had — but the agent was quietly less capable than it should be. That invisible failure is why retrieval recall is the metric to guard, and why description quality is an engineering deliverable, not an afterthought.