Scoring Agent Tool Calls Offline: Argument Normalisation, Equivalence Rules, Order-Free Parallel Calls and Credit for Abstaining

Why exact match lies in both directions

An offline evaluation of tool use usually starts with a set of prompts and the tool calls a correct agent would make. The first scorer anyone writes compares the produced call with the expected one as JSON. It reports a model that writes "city": "paris" instead of "Paris" as wrong, a model that lists two parallel lookups in the other order as wrong, and a model that omits an argument whose default is the expected value as wrong. The resulting number is low, noisy and mostly about formatting.

The second scorer anyone writes lower-cases every string and ignores missing arguments. Now a refund to ORDER-77 instead of order-77, which may be a different order in a case-sensitive system, is right, and a missing amount is right because the scorer ignored it. That number is high and wrong in the direction that matters. A useful scorer encodes, per tool and per argument, which differences are harmless. That knowledge belongs to whoever owns the tool, not to the evaluation team guessing from examples.

Normalisation rules belong to the tool

Put an argument specification next to each tool definition, versioned with it. Each argument names how values compare.

Two rules keep this honest. Identifiers are exact unless the system that resolves them is provably case-insensitive. And free-text arguments are not compared by the call scorer at all; if they matter, score them with a separate rubric so that a string comparison does not pretend to measure meaning. Provider features such as strict schema adherence for function calling reduce malformed calls but do not change this: a call can be perfectly schema-valid and still name the wrong order.

Calls, sets and plans

Agents can emit several tool calls in one turn. When the calls are independent, their order carries no meaning and the scorer should match them as a multiset: pair each expected call with an unused produced call of the same tool whose arguments are equal under the rules. When the calls depend on each other, the dependency is visible in the arguments (the second call uses an id returned by the first), so order is checked by the trajectory, not by position.

Some tasks have more than one correct plan. Refunding a known order can be one call, or a lookup followed by the refund. Represent the expectation as a list of acceptable plans and score against the best match. And some prompts need no tool at all; the Berkeley Function-Calling Leaderboard reports a separate relevance-detection category for exactly this. Abstention cases belong in every set, because an agent that calls a search tool for "thanks, that is all" wastes money and sometimes causes effects.

A scorer

Illustrative scorer, executed against the six cases below; dates are omitted for brevity.

def args_equal(spec, expected, produced):
    for name, rule in spec.items():
        a, b = expected.get(name, MISSING), produced.get(name, MISSING)
        if rule.get("default") is not None:
            a = rule["default"] if a is MISSING else a
            b = rule["default"] if b is MISSING else b
        if rule["kind"] == "ignored":
            continue
        if a is MISSING or b is MISSING:
            if a is not b:
                return False
            continue
        if rule["kind"] == "casefold" and a.casefold() != b.casefold():
            return False
        if rule["kind"] == "exact" and a != b:
            return False
        if rule["kind"] == "number" and abs(float(a) - float(b)) > rule.get("tol", 0):
            return False
        if rule["kind"] == "enum" and rule["aliases"].get(a, a) != rule["aliases"].get(b, b):
            return False
    return not (set(produced) - set(spec))     # unknown arguments fail

def plan_matches(specs, plan, produced):
    unused = list(produced)
    for want in plan:
        hit = next((c for c in unused if c["name"] == want["name"]
                    and args_equal(specs[want["name"]], want["args"], c["args"])), None)
        if hit is None:
            return False
        unused.remove(hit)
    return not unused                          # extra calls fail

def score(specs, plans, produced):
    if not plans:                              # the right answer is no tool call
        return not produced
    return any(plan_matches(specs, plan, produced) for plan in plans)

Worked example

Six cases from a support agent with two tools, get_order(order_id) and refund(order_id, amount, unit), where unit defaults to EUR and accepts the alias euro. Exact JSON comparison and the rule-based scorer above disagree on four of them.

Exact match scores 0 of 6 and the rule-based scorer 4 of 6. The two failures are real: an identifier with the wrong case and a needless call. A scorer that lower-cased everything would have passed case 3 and reported 5 of 6. Keep a small set of such adversarial cases in the scorer's own test suite, because a scorer is code that can regress.

When to stop scoring calls

Call-level scoring works for single turns and short plans. For multi-turn tasks with many valid paths it becomes a list of exceptions. The tau-bench authors evaluate by comparing the database state at the end of a conversation with an annotated goal state, which scores outcomes instead of paths and is a better fit wherever the tools write to a store you can snapshot. Use both: call scoring to diagnose which tool or argument is weak, state comparison to decide whether the agent did the job. Report name accuracy, argument accuracy and abstention accuracy separately, since an aggregate hides which one moved.

Calibrating the scorer

• Sample 100 cases where exact match and the rule-based scorer disagree and have a person label them; disagreements between the person and the scorer are scorer bugs.
• Version argument specifications with the tool; a renamed enum value silently turns passes into failures.
• Fail closed on unknown arguments and on extra calls, then relax only with a recorded reason.
• Re-run the scorer on stored outputs whenever its rules change, and keep the old number beside the new one for a release.

Failure modes

• Global lower-casing, which forgives identifier mistakes.
• Ignoring missing arguments instead of applying documented defaults.
• Position-sensitive comparison of independent parallel calls.
• No abstention cases, so needless calls never count against the agent.
• Scoring free-text arguments by string equality and calling the result a quality measure.

Practice and related reading

Further reading:

• Tool use failure modes: a taxonomy for traces and evals
• Tool calling best practices
• Custom LLM benchmarks from your own traffic

Sources and scope

Checked against these sources on 21 October 2026. Examples and figures are illustrative, not measurements.

• Berkeley Function-Calling Leaderboard blog: AST and executable evaluation categories
• Patil et al., Gorilla: Large Language Model Connected with Massive APIs (arXiv 2305.15334)
• Yao et al., tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (arXiv 2406.12045)
• OpenAI API documentation: Function calling (strict mode, parallel calls)
• Claude API documentation: How to implement tool use