Why architecture matters here
The architectural case for guided decoding is that validity is not a property you can reliably obtain by asking. Prompt engineering — 'respond only with valid JSON, no prose' — shifts the failure rate but never to zero, and the failures cluster exactly where they hurt: long outputs, unusual inputs, edge cases. Any system that pipes model output into a parser or an API is only as reliable as its worst response, and a 1% malformed rate at scale means thousands of broken calls a day. Guided decoding changes the guarantee from statistical ('usually valid') to structural ('always valid'), which is the difference between hoping and knowing.
That structural guarantee has an outsized effect on the systems built on top. Tool-calling and agent frameworks depend on the model emitting a function name and arguments that match a schema; if the arguments occasionally fail to parse, the agent stalls or takes a wrong action. Data-extraction pipelines that pull structured records from documents need every record to fit the target schema or the load fails. In both cases guided decoding removes an entire class of runtime error at the source, so the surrounding code no longer needs defensive parsing, retry loops, and repair heuristics — it can trust the output's shape absolutely and spend its complexity budget on logic instead.
There is also a cost story that runs the opposite direction from intuition. Constraining the sampler sounds like extra work, and it does add per-step mask computation, but it eliminates the retry loop, which is the dominant cost of unconstrained structured generation. A retry means running the whole generation again — full prompt, full output — sometimes several times. Guided decoding replaces those probabilistic multiple full generations with one guaranteed-valid generation plus a small per-token overhead. For any workload where malformed output was previously being retried, guided decoding is usually a net latency and cost win, not just a reliability one.
Finally, guided decoding reframes the boundary between the model's job and the system's job in a healthy way. The model is responsible for content — what the name should be, what the extracted value is, which tool to call — and the FSM is responsible for form — that the output is well-formed JSON matching the schema. Separating these concerns means you can reason about them independently: correctness of form is a property of the grammar you compiled, provable and testable, while quality of content remains a property of the model and prompt. Systems that blur the two, trying to get form correctness out of prompting, end up unable to guarantee either.
The architecture: every piece explained
The first component is the specification: what shape the output must take. This is expressed as a JSON schema, a context-free grammar, a regular expression, or a type definition — anything that precisely describes the allowed strings. The specification is the contract; everything downstream exists to enforce it. A JSON-schema-driven system, for example, expands the schema into a grammar that describes exactly the JSON documents satisfying it: the right keys, the right value types, required fields present, no extras.
That specification is compiled into a finite-state machine — an automaton whose states represent positions within the structure ('after the opening brace, expecting a key', 'inside a string value', 'expecting a comma or closing brace') and whose transitions say which characters or tokens legally move from one state to the next. Compilation is the expensive step, so it is done once per grammar and cached; a given schema is compiled the first time it is seen and the resulting automaton is reused for every request that shares it. The automaton is what lets the decode loop answer, cheaply and at every step, the question 'which tokens are legal right now?'
At generation time the mechanism is logit masking inside the decode loop. The model produces a logit for every token in the vocabulary at each step. Given the FSM's current state, you compute the set of tokens that are legal transitions and build a mask that sets the logits of all illegal tokens to negative infinity. After masking, the probability of any illegal token is exactly zero, so whatever sampling strategy runs next — greedy, top-p, temperature — can only select a legal token. The chosen token is then fed back to advance the FSM to its next state, and the loop continues until the automaton reaches an accepting end state.
The subtle, essential piece is tokenizer alignment. Grammars are naturally defined over characters, but the model emits subword tokens, and a single token may span a character boundary that matters to the grammar — one token could contain the closing quote of a string and the following comma. Guided-decoding implementations must therefore reason about which tokens, given the current FSM state, could partially or fully advance the automaton, precomputing for each state the set of vocabulary tokens that are legal. This token-level automaton (rather than a naive character-level one) is what makes masking a single fast set-lookup per step instead of an expensive re-parse, and getting it right is the difference between a correct, fast implementation and one that either rejects valid tokens or leaks invalid ones.
End-to-end flow
Walk a single guided generation producing a JSON object for the schema {name: string, age: integer}. Before decoding, the schema is compiled (or fetched from cache) into an FSM: the start state permits only {, the next expects a key string, and so on. The model receives the prompt and the decode loop begins. At the first step the model's logits might favor a friendly Sure or a newline, but the FSM's start state marks every token except those beginning the JSON object as illegal, so the mask zeroes them out and the only survivors are tokens that open the object. The sampler picks {. The model's chattiness is overridden by structure at the very first token.
The FSM advances to 'expecting a key.' Now legal tokens are those that continue toward a valid key — the quote and the letters of an allowed field name. The model, guided by its training and the prompt, samples toward "name"; at each step the mask keeps it on the rails, disallowing, say, a key the schema does not define. After the key comes the colon, then a string value: the FSM enters a 'string contents' state where nearly any character is legal until a closing quote, so here the model has wide latitude to emit the actual name — this is where content freedom lives. The model writes the person's name, the closing quote advances the FSM, and a comma leads to the next field.
The age field shows the constraint doing real semantic work. Its schema type is integer, so once the FSM is in the 'age value' state, the only legal tokens are digits (and structural terminators). If the model, influenced by the prompt, 'wants' to emit "unknown" or null, those tokens are masked to negative infinity and cannot be chosen; the sampler is forced to pick a digit. The output cannot contain a type error because the tokens that would create one were never candidates. This is guided decoding's core promise made concrete: not 'the model usually returns an integer for age' but 'the model cannot return anything but an integer for age.'
Finally the object closes. After the last field the FSM permits the closing brace; the mask allows it, the sampler emits }, and the automaton reaches an accepting state, which also serves as a natural stopping signal. The generation ends with a string that is guaranteed to be valid JSON matching the schema — no trailing prose, no missing brace, no wrong type — and it did so in a single pass with no parse-and-retry. Contrast the unconstrained path: the same model might have produced Here is the JSON: {name: ... with a leading sentence and an unquoted key, failed the parser, and triggered a full regeneration. Guided decoding spent a small mask computation per token to make that failure impossible.