Why architecture matters here
Small models are deployed precisely where reliability matters most and where the model has the least slack to spare. An SLM running on a phone or an edge box is doing structured work — extracting fields, choosing a tool, emitting a command — and the thing consuming its output is a parser, not a human who can forgive a missing brace. A single malformed token means the parse fails, and the usual recovery (retry the whole generation) is expensive on constrained hardware and still probabilistic. The reliability of the format is therefore a first-class product requirement, not a nice-to-have.
And formatting is exactly where small models are weakest. A 70B model has enough capacity to internalize 'when I am asked for JSON I close every brace and quote every key' as a near-reflex; a 2B or 3B model does not, and it will, some meaningful fraction of the time, drop a comma, hallucinate a trailing field, emit prose before the JSON, or wander off the schema. Prompt engineering — 'respond ONLY with valid JSON matching this schema' — reduces the rate but never drives it to zero, and 'never' is what a downstream parser needs. Fine-tuning helps but costs data and a training run and still leaves a tail. The economics of SLM deployment make an approach that guarantees validity, rather than merely improving the odds, extremely valuable.
Guided decoding provides that guarantee by changing where the constraint lives. Instead of hoping the model's learned distribution stays inside the valid set, you make the invalid set unreachable at sampling time. The validity is now a property of the decoding harness, independent of the model's size or training — a 1B model with guided decoding produces JSON that parses as reliably as a 200B model, because neither is allowed to emit a token that would break the parse. For small-model deployments this decouples format reliability from raw capability, and that decoupling is the whole point.
There is a subtler benefit that matters on constrained hardware: guided decoding can make generation cheaper, not just safer. When the grammar forces a long stretch of the output — the fixed keys of a JSON object, the punctuation between fields, a closing bracket — there is often exactly one legal token, and a decoder that knows this can emit those tokens directly without spending a full forward pass choosing among a masked vocabulary. Even when it does run the model, the constraint prunes the search so aggressively that the effective work per structural token collapses. So the technique that guarantees a parseable output can also shorten the critical path, which is why edge and agent deployments reach for it not only to be correct but to be fast.
The architecture: every piece explained
Top row: the compilation pipeline, done once per schema. The schema or grammar is the specification of valid output — a JSON Schema, a regular expression, or a full context-free grammar (EBNF). It is compiled to a finite-state automaton (an FSA, or for a CFG a pushdown automaton): a set of states with allowed transitions, where each state encodes 'given what we have emitted so far, here is what may legally come next.' Because the vocabulary is fixed, the compiler precomputes a token-to-transition map for each state — which of the tens of thousands of vocabulary tokens keep the output valid from here. This is the expensive step, and it is amortized across every generation that uses the schema.
Middle row: the per-step decoding loop. The model produces its logits — a score for every token in the vocabulary. The harness looks up the current FSA state's allowed-token mask and applies it: every token that is not a legal next token has its logit set to negative infinity, so its post-softmax probability is exactly zero. The model then samples from the surviving (renormalized) distribution — still using its own learned preferences among the valid options — and the harness advances the FSA to the state that the chosen token transitions to.
Bottom rows: termination and output. The loop repeats until the FSA reaches an accept state, at which point the output is a complete, valid string by construction — there was no path to an invalid one. The structured output is therefore guaranteed to parse against the original schema, every time. At runtime the mask for the current state is built (or looked up) each step and fused into the sampler alongside the usual temperature and top-p logic, so the constraint is just one more transformation of the logits before sampling and adds only the cost of an already-computed mask.
The distinction between a plain FSA and a pushdown automaton matters for anything with nesting, which is most real schemas. A regular expression or a flat set of enum values compiles to a finite-state automaton, but JSON is recursive — objects contain objects, arrays contain arrays — and recursion cannot be captured by a fixed set of states alone. A context-free grammar compiles instead to a pushdown automaton, which augments the state machine with a stack: opening a brace pushes a 'expecting to close this object' obligation, and the closing brace pops it, so the automaton tracks nesting depth exactly. The practical consequence is that the allowed-token mask at any step depends not just on a flat state but on the top of the stack, and a correct implementation must compute it from both. Good engines precompute as much of this as possible — caching the per-state token sets and updating only the stack incrementally — so that even a deeply nested schema costs little more per step than a flat one, keeping the guarantee affordable on the small hardware SLMs run on.
End-to-end flow
Walk through generating a response to a schema requiring {"sentiment": "positive"|"negative", "score": number}. The schema compiles offline into an FSA whose start state permits exactly one token: the opening brace. At step one the model produces logits over the full vocabulary — it might, left unconstrained, prefer to start with the word 'Sure' — but the mask zeroes everything except {, so { is emitted regardless, and the FSA advances to the 'expecting-a-key' state. That state permits only the tokens that begin the quoted key "sentiment", so the model is walked through those tokens deterministically.
Now a genuine choice appears. After {"sentiment": the FSA is in a state permitting the start of either "positive" or "negative". Here the mask allows two branches, and the model's own preference decides — it samples according to its learned distribution over just those two valid continuations. Suppose it picks 'positive'; the FSA follows that branch, emits the remaining tokens of the value, and moves to the state expecting a comma. The comma, the "score" key, and the colon are again forced. At the number position the FSA permits the token classes that form a valid JSON number — digits, a decimal point, a minus sign in the right place — and the model chooses the actual value freely within that grammar. Finally the FSA permits the closing brace, reaches its accept state, and generation stops with a string that is provably valid JSON matching the schema.
The efficiency payoff is visible in that trace. Long runs of the output — the braces, the key names, the punctuation — had exactly one legal token, so a well-built decoder can emit them without agonizing over the full vocabulary, and some implementations skip the forward pass entirely for these forced tokens (a form of grammar-driven jump-ahead). The model's expensive attention only truly earns its keep at the branch points — the sentiment choice, the numeric value — where there is a real decision to make. On a small model running on modest hardware, this concentration of work onto the few genuine decisions is a material speedup on top of the correctness guarantee.
Contrast the unconstrained path to see what was bought. Without guidance, the same 2B model asked for this JSON might emit Sure! Here is the result: {"sentiment":"positive","score":0.9} — the leading prose alone breaks a strict parser, and on a bad draw it might also forget the closing brace or invent a third field. The downstream service then retries, doubling latency and cost, and still with no guarantee. Guided decoding removes that entire failure class: the leading prose was never sampleable because the start state only allowed a brace, the extra field was never sampleable because the FSA had no transition for it, and the missing brace was impossible because the accept state requires it. The output is correct on the first try, which is why extraction pipelines and tool-calling agents built on small models lean on this rather than on retry loops. The same reasoning extends to tool-calling: when an agent must choose one of a fixed set of function names and emit arguments matching that function's parameter schema, guided decoding compiles the union of tool signatures into the grammar, so the model cannot invent a nonexistent tool or pass a malformed argument object — the invalid call is simply not on any path the automaton will walk.