SLM function calling architecture, in depth: tool registry, retrieval, constrained decoding, validation, authorization and escalation

The contract: what the model owns

Start with what the model and the harness each own. The model owns intent: which tool, if any, fits the request, and what argument values the user meant. The harness owns everything that can be checked mechanically. That includes output syntax, whether the tool exists, argument types and ranges, cross-field rules, permissions, execution limits, retries and the decision to give up. Every responsibility you leave with a small model is one it will occasionally fail, so move as much as possible into code.

The contract is the tool registry. Each entry has a name, a one-line description written for the model, a JSON Schema for the arguments, and metadata the model never sees: required permission scope, whether the call has side effects, whether it needs user confirmation, a timeout and an idempotency policy. Descriptions are prompt text and deserve the same care. Small models lean heavily on the first sentence, so state when to use the tool and when not to.

REGISTRY = {
  "set_timer": {
    "description": "Start a countdown timer. Not for alarms at a clock time.",
    "schema": {"type": "object", "additionalProperties": False,
               "required": ["minutes"],
               "properties": {"minutes": {"type": "integer", "minimum": 1, "maximum": 720},
                              "label": {"type": "string", "maxLength": 40}}},
    "scope": "timers", "side_effects": True, "confirm": False, "timeout_s": 2,
  },
  "send_message": {
    "description": "Send a text message to a saved contact.",
    "schema": {"type": "object", "additionalProperties": False,
               "required": ["contact_id", "text"],
               "properties": {"contact_id": {"type": "string", "pattern": "^c_[0-9]+$"},
                              "text": {"type": "string", "maxLength": 500}}},
    "scope": "messages", "side_effects": True, "confirm": True, "timeout_s": 5,
  },
}

Shrinking the choice with retrieval

Small models degrade sharply as the tool list grows: more schemas mean more context, more similar names to confuse and less attention for the user's words. Retrieval fixes that before decoding starts. Embed each tool's description and a few example requests once. At each turn, embed the user message, take the top five to eight tools by similarity, and always add a few core tools, such as asking a clarifying question, so the model is never forced into a bad fit.

Measure retrieval separately, as recall at k: how often the correct tool is in the shortlist. If the right tool is missing, no amount of decoding cleverness helps. Log every case where the model asks for clarification or escalates and check whether retrieval failed first. Most "the model is dumb" bugs in these systems turn out to be retrieval bugs.

Prompting and constrained decoding

Next, render the shortlist into the prompt in the model's own format. Hugging Face tokenizers accept apply_chat_template(messages, tools=tools, add_generation_prompt=True) for models whose chat template supports tools. Use that rather than hand-writing a format, because a model fine-tuned on one tool-call syntax performs noticeably worse on another.

Then constrain decoding. Build one JSON Schema that says the output is exactly one of the shortlisted calls or a plain reply, and pass it to a runtime that masks every token that would break it. With llama.cpp's server you can send the schema in the request (json_schema, or a hand-written GBNF grammar). vLLM, SGLang and Outlines offer the same capability, but vLLM's parameter names have changed between releases, so check the documentation for the version you deploy rather than copying an old example. The schema makes unknown tool names and missing required fields impossible to emit.

def call_schema(shortlist):
    calls = [{"type": "object", "additionalProperties": False,
              "required": ["name", "arguments"],
              "properties": {"name": {"const": n},
                             "arguments": REGISTRY[n]["schema"]}} for n in shortlist]
    reply = {"type": "object", "additionalProperties": False, "required": ["reply"],
             "properties": {"reply": {"type": "string"}}}
    return {"oneOf": calls + [reply]}

def decode(prompt, shortlist):
    r = requests.post(LLAMA_URL + "/completion", timeout=30, json={
        "prompt": prompt, "json_schema": call_schema(shortlist),
        "temperature": 0.0, "n_predict": 256})
    return json.loads(r.json()["content"])

Two warnings. Always include the plain-reply branch. A schema that only allows tool calls forces a call on every turn, including "thanks!", and the model will pick the least bad tool. Also, constraints shape the distribution as well as the syntax. When the grammar forbids the token the model wanted, it takes the next allowed one, which can be valid and wrong. That is why validation still matters. Background on output constraints for small models is in structured output with SLMs.

Validation and the repair loop

Constrained output is syntactically valid but can still be wrong. Validate in two layers. The first re-checks the schema with a standard validator, because grammar engines support different subsets of JSON Schema and silently ignore some keywords. The second applies semantic rules a schema cannot express. Does the contact ID exist? Is the end date after the start date? Is that amount inside the user's limit? Did the argument actually come from the conversation? The last check catches the most common small-model fault, invented identifiers, by requiring that IDs appear in the session state or in a previous tool result.

from jsonschema import Draft202012Validator

def validate(call, session):
    if "reply" in call:
        return None
    spec = REGISTRY[call["name"]]
    errs = [e.message for e in Draft202012Validator(spec["schema"]).iter_errors(call["arguments"])]
    args = call["arguments"]
    if call["name"] == "send_message" and args.get("contact_id") not in session.known_ids:
        errs.append(f"contact_id {args.get('contact_id')!r} not found; call find_contact first")
    return "; ".join(errs) or None

def run_turn(messages, session, max_repairs=1):
    shortlist = retrieve(messages[-1]["content"])
    for attempt in range(max_repairs + 1):
        call = decode(render(messages, shortlist), shortlist)
        if "reply" in call:
            return call["reply"]
        err = validate(call, session)
        if err is None:
            return authorize_and_execute(call, session)
        messages = messages + [
            {"role": "assistant", "content": json.dumps(call)},
            {"role": "user", "content": f"Your tool call was invalid: {err}. Fix it."}]
    return escalate(messages, reason=err)

Keep the repair budget at one attempt. Small models that fail twice on the same turn rarely succeed on the third try, and every retry adds latency. The repair message should be short and specific, naming the field and the rule, because the model will copy whatever you say.

Authorization and safe execution

Authorization is decided by the harness from registry metadata and session identity, never by the model. Check the tool's scope against the user's grants. Require explicit user confirmation for tools marked confirm, showing the exact arguments. Apply per-tool rate limits so a looping model cannot send fifty messages. Run each tool with its own timeout and catch its exceptions, returning a structured error to the model rather than a stack trace.

Retries and repairs create a duplication risk. If a side-effecting call times out after it actually ran, a naive retry runs it twice. Attach an idempotency key derived from the turn ID and call content, and have tools reject a repeated key. Finally, treat tool results as untrusted data. A web page or email returned by a tool can contain instructions aimed at the model. Wrap results in a clearly delimited data block, strip anything resembling the tool-call format, and never let a tool result widen the permissions of the next call. Device-side tool exposure through MCP is discussed in MCP on edge devices.

Escalation and metrics

Escalation is what makes a small model safe to deploy. When repair fails, when retrieval confidence is low, or when the request needs more than a set number of chained calls, hand the turn to a larger model or ask the user a clarifying question. Decide by policy what may leave the device. A privacy-sensitive assistant may prefer asking the user over sending the conversation to a cloud model. Track the escalation rate as a first-class metric. If it rises after a tool is added, the new tool's description or schema is probably confusing the model.

The metrics that matter, measured on a held-out set of real requests with expected calls: retrieval recall at k, exact tool-choice accuracy, argument accuracy per field, the rate of correct no-call replies, the repair success rate, the escalation rate and end-to-end latency at p95. Running models locally is covered in on-device LLM inference.

Worked example: a phone assistant

A phone assistant uses a 3B model running locally with about 40 tools: timers, alarms, messages, calendar, notes and settings. A first prototype with all 40 schemas in the prompt and no grammar produced parse failures on a noticeable share of turns and confused set_timer with set_alarm constantly. The rebuilt harness retrieves six tools per turn plus ask_user, constrains output to the shortlist or a reply, validates IDs against session state and repairs once.

Replaying a set of recorded requests through the new harness shows a pattern you should expect. Parse failures disappear, because the grammar rules them out. Most remaining errors are wrong argument values, mainly relative times such as "in an hour" against "at one". The fix is a deterministic tool, resolve_time, that the model calls instead of doing date arithmetic itself. Escalations concentrate in multi-step requests. This is an illustration, not a benchmark, so build your own evaluation set before trusting anyone's numbers.

Failure modes

Trade-offs

Every layer costs something. Retrieval adds an embedding call and a new failure point, so a catalogue of five tools may not need it. Constrained decoding adds grammar compilation time, which runtimes cache per schema, so keep shortlist schemas stable and cacheable. Strict schemas reject odd but valid requests, and loose schemas let nonsense through. Tighten the fields that carry risk, such as IDs, amounts and recipients, and leave free text free. A repair attempt roughly doubles latency for the turns that need it. Escalation buys accuracy with privacy and cost. Fine-tuning the model on your own tool format reduces how often the harness has to step in, but it does not remove the need for the harness, because validation, authorization and idempotency protect you from failures no amount of training eliminates.

What to do next

1. Write the registry with schemas, descriptions and the metadata the model never sees: scope, side effects, confirmation, timeout.
2. Build a held-out evaluation set of real requests, including ones that should produce no call.
3. Add retrieval and measure recall at k before tuning anything else.
4. Constrain decoding to the shortlist plus a reply branch, using your runtime's documented schema option.
5. Re-validate every call and add semantic checks, starting with identifier provenance.
6. Enforce authorization, confirmation, rate limits and idempotency in the harness.
7. Allow one repair, then escalate, and track the escalation rate per tool.