Catch Tool Selection Regressions Before Editing Agent Tool Descriptions

A description edit is a prompt change

Every tool an agent can call reaches the model as a name, a JSON schema and a description. The model reads the description to decide which tool fits the request, so the description is part of the prompt. The Claude tool-use documentation calls detailed descriptions the most important factor in tool performance, and OpenAI's function-calling guide asks for clear, detailed descriptions and names. Yet descriptions usually live next to the code, are edited in the same pull request as a bug fix, and are reviewed as documentation.

The help-desk agent at Belgavi's Book Store has four tools: lookup_order, start_return, issue_refund and check_stock. Customers often say "money back" when they mean a return, so someone proposes one helpful sentence for start_return: "Use this for anything about getting money back." This article measures what that kind of edit does, and builds the test that should run before it ships.

The test: fixed cases, a fixed runner, per-case results

A selection test needs three things. A case set: realistic requests with the tool a correct agent calls first, covering every tool and the boundaries between them. A runner that sends each case with the candidate tool set and records the first tool call, or none. And a comparison that reports which cases changed, not only the total.

I wrote 16 synthetic help-desk messages, four per tool, and ran each three times against two versions of the descriptions and then a third, deliberately broader one. The runs used Ollama 0.40.12 on one Windows machine, through its /api/chat endpoint with a tools list, temperature 0 and a fixed seed, with qwen2.5:1.5b and qwen2.5:0.5b-instruct. These are small local models chosen because they run offline. No hosted model was tested, so these results do not establish how one would compare; the same case-based test can be applied to other models.

Python, simplified from the probe used for the measurements below; post_json is a small urllib helper.

def first_tool(tools, text):
    body = {"model": MODEL, "stream": False, "tools": tools,
            "options": {"temperature": 0, "seed": 7},
            "messages": [{"role": "system", "content": SYSTEM},
                         {"role": "user", "content": text}]}
    reply = post_json("http://127.0.0.1:11434/api/chat", body)["message"]
    calls = reply.get("tool_calls") or []
    return calls[0]["function"]["name"] if calls else None

def diff(baseline, candidate):
    """Per-case changes between two runs: {text: (old_tool, new_tool)}."""
    return {t: (baseline[t], candidate[t]) for t in baseline if baseline[t] != candidate[t]}

What the runs showed

Three things stand out. First, the sentence I expected to break refund routing changed nothing on the 1.5b model; my prediction about the edit was wrong, which is the argument for measuring instead of reviewing. Second, on the 0.5b model the same edit raised the total from 34 to 38 while adding a failure on a stock question, a tool whose description nobody touched. An accuracy number would have approved that change. Third, the broader v3 edit moved "I already mailed back BK-1047 last week, where is my refund?" and "You charged me twice for BK-1055, please refund one charge." to start_return on every repeat: a clean, reproducible regression, exactly the kind a release gate should block.

At temperature 0 with a fixed seed, repeats agreed on every case for the 1.5b model and on all but one or two cases for the 0.5b model. Hosted APIs are less repeatable than that, so run each case several times and treat a case that changes across repeats as unstable rather than as a regression.

Reading a diff instead of a score

• New failures block, fixed cases do not compensate. A change that fixes two cases and breaks one is a trade to decide on, not an improvement to merge.
• Look at the destination. A refund request landing in start_return is a customer annoyance; one landing in issue_refund without a return is money. Weight cases by the cost of the wrong tool.
• Watch "no tool". A request that calls nothing usually gets a chatty answer instead of an action, which looks fine in logs.
• Keep boundary cases. The useful cases sit between two tools: "money back" for a delivered book (return) and for a lost parcel (refund).

Versioning descriptions

Treat the tool set as an artifact: names, schemas and descriptions together, with a version and a content hash recorded on every agent run. When a run misbehaves you can then tell which descriptions it saw. Store the description text in one place that both the agent and the test runner read, so the test cannot drift from what ships. When tools come from an MCP server you do not control, record the server's tool list hash per run and re-run the selection test when it changes.

Failure modes of the test itself

• Cases written from the descriptions. If the person who edited the description writes the cases, the cases echo its wording. Take cases from real, anonymised requests.
• One model only. The 0.5b and 1.5b models disagreed about the same edit. Test with the model you ship, and re-run when the model changes.
• System prompt drift. The runner must send the same system prompt and tool order as production; both change selection.
• Too few cases per boundary. Four cases per tool found these effects, but a boundary with one case can flip on noise.

Writing boundary cases

The most useful cases are pairs that differ in one fact and should go to different tools. "My copy of BK-1044 arrived damaged, I want my money back" is a return; "BK-1046 never arrived, I want my money back" is a refund. A description edit that blurs the boundary moves one of the pair. Write at least two such pairs for every boundary the support team argues about, and add every misrouted production request as a new case once it has been anonymised. Cases that are easy for every version, such as a plain stock question, still belong in the set, because the 0.5b run above shows that an edit elsewhere can break them.

Where this fits

Run the selection test in CI whenever a tool definition, the system prompt or the model changes, and block on new failures in high-cost cases. It complements end-to-end evaluation rather than replacing it: it is cheap, fast and points at the exact case that moved, while a task-level evaluation tells you whether the agent still finishes the job. The companion example in agent-decision-lab contains the case set, the runner, recorded outputs from these runs and an offline diff that runs without a model.

Practice and related reading

Further reading:

• Tool selection architecture for agents with hundreds of tools
• Scoring Agent Tool Calls Offline
• MCP tool annotations architecture

Sources and scope

Checked against these sources on 3 November 2026. Measured on 3 November 2026: Ollama 0.40.12 on one Windows 11 machine, qwen2.5:1.5b and qwen2.5:0.5b-instruct, temperature 0, seed 7, 16 synthetic cases, three repeats each. Small local models; results on other models will differ.

• Claude documentation: How to implement tool use (writing tool descriptions)
• OpenAI documentation: Function calling (best practices for defining functions)
• Ollama documentation: Tool calling