Why architecture matters here
The naive approach — paste tool descriptions into the prompt and hope the SLM emits parseable JSON — fails at rates that make products unusable. A 3B model prompted with ten tool schemas will produce syntactically broken JSON in a meaningful fraction of calls, and semantically wrong calls (real function, wrong or missing arguments) far more often. Each failure either surfaces to the user as a non-answer or triggers an expensive blind retry with no better odds.
Architecture matters because every layer of the stack can convert a class of model error into a non-event. Constrained decoding eliminates the entire category of malformed output: if the sampler can only choose tokens permitted by a JSON-schema grammar, the model physically cannot emit a syntax error or an unknown function name. Validation plus repair converts semantic errors from user-visible failures into one extra low-latency inference. The fallback router converts the remaining hardcases into slightly slower correct answers instead of wrong ones. The same 3B model, wrapped this way, moves from unusable to production-grade call accuracy on a bounded tool catalog.
The economics compound. An on-device SLM call costs effectively nothing and returns in tens to hundreds of milliseconds; a cloud LLM call costs real money and hundreds of milliseconds to seconds. If the harness lets the small model safely handle even 80% of tool invocations, the blended cost and latency of the system drop by most of that fraction — and the private requests that never leave the device are a compliance feature you cannot buy any other way. The architecture is what makes the 80% safe.
The architecture: every piece explained
Prompt assembler. Tool schemas live in a registry — name, description, JSON-schema parameters, auth scope — and the assembler renders the subset relevant to this turn into the model's native tool-prompt format. Selection matters for small models: a 3B model reasons noticeably better over 5 candidate tools than 40, so a cheap embedding retrieval step often pre-filters the catalog before the model ever sees it. Few-shot examples of correct calls, in exactly the output format the grammar will enforce, are the highest-leverage tokens in the prompt.
Grammar-constrained sampler. The registry's JSON schemas are compiled into a decoding grammar (GBNF in llama.cpp, regex/CFG guides in vLLM and Outlines, token-level FSMs in TensorRT-LLM). At each decoding step the runtime intersects the model's logits with the set of tokens that keep the output inside the grammar — everything else is masked to negative infinity. Function names become an enum over the registry, so hallucinated tools are unrepresentable; required fields become mandatory productions, so they cannot be omitted. Compiled grammars are cached per tool-set hash because compilation is milliseconds but not free.
Streaming parser and validator. Structural validity is guaranteed; semantic validity is not — the grammar cannot know that end_date must follow start_date or that a city name must resolve. An incremental JSON parser consumes tokens as they stream, so the validator can reject a bad enum value or an out-of-range number the moment it closes, cutting the failed decode short instead of paying for the full sequence.
Repair loop and router. On validation failure, the harness re-prompts the same SLM with the original context plus a terse machine-readable error (“value ‘tomorow’ for field date does not match format YYYY-MM-DD”). One repair round fixes the majority of semantic errors; a bounded budget (typically two) caps latency. Past budget — or when a confidence signal like low sequence log-probability fires — the router escalates the turn to a larger model. Executor and policy guard sit after the model: allow-listed functions, per-tool timeouts, argument audit against the caller's auth scope, and sandboxing for anything that touches the filesystem or network.
End-to-end flow
Follow one request through: a user tells an on-device assistant “move my 3pm with Priya to Thursday morning.” (1) The assembler retrieves the top-k relevant tools — calendar.search, calendar.update_event, contacts.resolve — and renders their schemas plus two few-shot examples into the prompt. The grammar for exactly these three schemas is fetched from cache. (2) The SLM decodes under the grammar. The first structural choice the grammar permits is the function-name enum; the model's logits favor calendar.search, and every subsequent token is constrained to that function's parameter schema. It emits {"name":"calendar.search","arguments":{"query":"Priya","date":"2026-07-09","time":"15:00"}}.
(3) The streaming parser validates fields as they close — date format, time format, no unknown keys — and the call passes. The policy guard confirms calendar.search is read-only and in scope. The executor runs it with a 2-second timeout and returns one matching event. (4) The result injector serializes the observation into the conversation as a tool-result message, and the loop re-enters the model for the next step. (5) The model now calls calendar.update_event with the event ID and a new start time. Suppose it emits "new_start":"Thursday 9am" — structurally legal (it is a string) but semantically wrong for a field declared as ISO-8601. The validator rejects it with a machine-readable error; the repair prompt appends the error and the model, now anchored, emits "2026-07-16T09:00:00". Total cost of the mistake: one extra 200ms decode, invisible to the user.
(6) Because update_event is a mutating call, the policy guard requires confirmation policy: the harness surfaces “Move ‘Sync with Priya’ to Thu 9:00?” and executes on approval. (7) Every step — prompt hash, grammar hash, raw model output, validation verdicts, repair count, executor latency — lands in a structured trace. That trace is the dataset for the next fine-tuning round: real failures become training examples, and call-accuracy dashboards are computed directly from it. The loop between production traces and fine-tunes is what makes a small model keep improving on your tool catalog specifically.