Constrained Decoding: How JSON Schema Mode Masks Invalid Tokens

Why prompting alone never reaches zero

A language model emits one token at a time by sampling from a probability distribution over its whole vocabulary, often 100,000 tokens or more. Asking for JSON in the prompt raises the probability of tokens that continue valid JSON, but it leaves every other token with some probability. Over a 60-token object, small per-token risks compound. If each generated token has a 0.06% chance of being a syntax slip, the chance of at least one slip is 1 - (1 - 0.0006)^60, about 3.5%. Add occasional prose before the object ("Here is the JSON:"), a wrong type in one field, or a missing required key, and an endpoint that looks reliable in a demo fails validation on several percent of production requests.

The lab's default setup is a prompt-only extractor with eight fields at temperature 0.7. Its synthetic model fails validation on about 13% of requests, with failures split between syntax slips, schema mismatches and prose before the JSON. Lowering the temperature helps, but only to about 10%. Mission one asks you to get it above 99.5% valid.

From schema to grammar to mask

Constrained decoding adds one step to the decode loop. Before sampling, the engine asks a grammar which tokens may come next given everything generated so far, and sets the logit of every other token to minus infinity. Softmax then assigns them zero probability and renormalises over the rest:

state = grammar.initial_state()
while not grammar.is_final(state):
    logits = model.forward(tokens)                 # one decode step
    allowed = grammar.allowed_tokens(state)        # bitmask over the vocabulary
    logits[~allowed] = float("-inf")
    token = sample(softmax(logits / temperature))
    state = grammar.advance(state, token)
    tokens.append(token)

The grammar comes from the schema. Regular structures (enums, numbers, fixed key order) compile to a finite-state machine; nested objects and arrays of arbitrary depth need a pushdown automaton, which is how context-free grammars are recognised. The hard part is that grammars are defined over characters while models emit tokens, and a single token can span several grammar symbols, for example ":" or "},. The engine must know, for each grammar state, which of the vocabulary's tokens keep the text inside the language.

The mask changes the distribution in a precise way. Suppose at the first output position the model assigns 0.6 to {, 0.3 to a code fence and 0.1 to prose. In schema mode only { survives, and after renormalisation it has probability 1. That is the easy case: the model's preferred token was valid. The hard case comes later.

JSON mode and schema mode are not the same thing

Many APIs and servers offer two levels. JSON object mode guarantees syntactically valid JSON but says nothing about keys or types. Schema mode compiles the specific schema and guarantees that the output validates against it, including required keys, types, enums and, depending on the implementation, numeric ranges and string patterns.

Schema support is rarely complete. Engines typically support a subset of JSON Schema, and features such as complex oneOf combinations, pattern properties or recursive references may be unsupported, rejected or silently ignored. Hosted strict modes often add their own rules, for example requiring every property to be listed as required and forbidding additional properties, so optional fields are expressed as a union with null. Check the documentation of the engine and version you deploy, and test that a deliberately wrong schema is rejected rather than ignored.

The cost: compilation and per-token mask work

Computing the mask naively means checking every vocabulary token against the grammar at every step. With a 128,000-token vocabulary, even 10 nanoseconds per check is about 1.3 ms per generated token, a sizeable fraction of a 10 to 20 ms decode step, and it runs on the CPU while the GPU waits unless the engine overlaps the two.

Implementations take different routes. Outlines, described by Willard and Louf, compiles a regular expression into a finite-state machine and precomputes an index from each state to its allowed tokens, so the per-token cost is a lookup but the compile step must visit every state and token; for large schemas that can take from hundreds of milliseconds to seconds. XGrammar, described by Dong and colleagues, uses a pushdown automaton and splits tokens into context-independent ones, whose validity it precomputes per grammar position, and context-dependent ones it checks at runtime, and it overlaps mask computation with GPU execution. Its authors report large speedups over earlier engines. Other engines, such as llguidance and llama.cpp's GBNF grammars, make their own trade-offs.

Compilation cost matters most when schemas change. If every request uses the same schema, the compiled grammar is cached and only the first request pays. If each request carries a different schema, as with per-user tool definitions, every request pays, and p95 time to first token rises by the compile time. Mission two in the lab sends a new 16-field schema with every request and asks you to bring p95 latency back under 2.2 seconds.

Jump-forward decoding: skipping tokens the grammar already knows

Much of a JSON object is fixed by the schema: braces, key names, colons, quotes and commas. When the grammar allows exactly one continuation, there is no need to ask the model. SGLang popularised jump-forward decoding with a compressed finite-state machine: runs of forced text are appended directly, and the model is only called where it actually has a choice. The appended text still has to enter the KV cache, but processing several known tokens at once is much cheaper than generating them one by one.

The gain depends on the schema. An object with many short fields, such as a tool call, is mostly structure, and in the lab's eight-field object about two thirds of tokens are forced. An object dominated by long free-text strings is mostly content, and the gain shrinks. One subtlety: forced text must be re-tokenised carefully, because appending characters can produce a token sequence the model would never have produced itself, which slightly shifts its predictions. Engines handle this boundary differently, so check how yours does it.

Valid but wrong: when the mask forces a value

Here is the hard case. A field is declared as a number, but the input contains no price. Unconstrained, the model would write null or a phrase like "not stated" with, say, 90% probability, and the output would fail validation, which is a visible error you can handle. In schema mode, both continuations are masked. The remaining 10% of probability is spread over digit tokens; after renormalisation, the model must write a number, and which number is close to arbitrary. The output validates, the pipeline succeeds, and the database now holds an invented price.

The fix is in the schema, not the decoder: declare fields that may be absent as nullable (for example a type union of number and null) or add an explicit status field. The same reasoning applies to enums without an "other" or "unknown" option, and to string fields with patterns the true answer cannot match. Research such as "Let Me Speak Freely?" by Tam and colleagues also found that strict format constraints can reduce reasoning quality on some tasks; a common mitigation is to let the model write its reasoning in a free-text field placed before the answer fields, or to reason first and format second. Mission three turns on missing facts and asks you to drive invented values below 0.5%.

Truncation defeats every mask

A grammar can force the next token to be valid, but it cannot make the output finish. If generation stops at max_tokens halfway through a string, the result is invalid JSON regardless of mode. This is the most common way teams see invalid output from a constrained endpoint, and it gets worse when inputs change: a feed of longer product descriptions or a new field can push typical objects past the limit. Size max_tokens from the longest objects you expect, with a margin, and alert on the share of requests that stop for length rather than end-of-sequence. Raising the limit costs nothing for short outputs, but the extra tokens on long ones cost latency, which is where jump-forward earns its keep. Mission four combines both pressures.

Tool calls, agents and retries

Tool calling is the largest consumer of constrained decoding in practice. An agent framework hands the model a list of tool definitions, each with a JSON schema for its arguments, and expects a well-formed call back. A malformed call is worse than a malformed answer, because the agent loop usually retries, and every retry is another full model call with a growing context. If an agent makes ten tool calls per task and each call has a 2% chance of being malformed, about 18% of tasks hit at least one retry (1 - 0.98^10). Constraining tool arguments to their schemas removes that whole class of retries.

Two cautions apply. First, constraining arguments does not make the choice of tool correct; a valid call to the wrong tool passes every check. Second, tool schemas are often generated per user or per session, which is exactly the dynamic-schema case where compilation cost and cache behaviour decide latency. Measure retries per task before and after, and keep validating arguments on the tool side anyway, because a schema-valid argument can still be out of range for the business rule behind the tool.

Knobs in real systems

The concepts are stable; names move between versions. Check your version's documentation for each of these:

• OpenAI-compatible APIs and vLLM: a response_format of type json_schema on the chat completions endpoint, with older vLLM versions also accepting extra parameters such as guided_json; the server chooses or is told which grammar backend to use (for example XGrammar, Outlines or llguidance).
• SGLang: JSON schema, regex and EBNF constraints, with grammar backend selection and jump-forward behaviour depending on version.
• Libraries: Outlines and XGrammar can be used directly with Hugging Face Transformers; llama.cpp accepts GBNF grammars and JSON schemas.
• Hosted strict modes: usually require additionalProperties: false and all fields listed as required; express optional values as nullable.

A practical example for the hypothetical Belgavi's Book Store, which extracts structured listings from supplier emails:

{
  "type": "object",
  "properties": {
    "title":    {"type": "string"},
    "price":    {"type": ["number", "null"]},
    "format":   {"enum": ["hardcover", "paperback", "ebook", "unknown"]},
    "in_stock": {"type": ["boolean", "null"]}
  },
  "required": ["title", "price", "format", "in_stock"],
  "additionalProperties": false
}

Every field is required, so the structure is guaranteed, but price and stock may be null and the format enum has an explicit escape value, so the model is never forced to invent an answer the email does not contain. Downstream code then treats null as "ask a human" instead of trusting a fabricated number.

Detecting problems in production

• Finish reasons. Count outputs that ended for length; with a mask in place, these are your remaining invalid outputs.
• Grammar compile time and cache hit rate. If your server exposes them, alert when compile time appears in TTFT percentiles.
• Value distributions. Invented values show up as unusual distributions: many prices with suspiciously round values, or an enum's first option chosen far more often after a schema change. Compare field distributions before and after enabling schema mode.
• Null rates. After making fields nullable, track how often each is null; a sudden jump is often an upstream input problem.
• Schema rejection. Test that unsupported schema features fail loudly in staging.

The common mistakes are treating schema-valid as correct, leaving max_tokens at a default sized for chat, compiling a fresh grammar for every request without caching, and writing schemas that give the model no honest way to say "not present".

Run the lab: Mask the logits: JSON schema mode without invented values

The lab samples 300 synthetic extraction requests. Each row on the canvas is one output drawn token by token: grey tokens are forced by the grammar, blue tokens are chosen by the model, and amber marks a number the mask forced the model to invent. The strip below shows a vocabulary mask, with allowed tokens lit. Start in prompt-only mode, then work through the missions: parse failures, a new schema on every request, missing facts and long descriptions.

Prefer a full-width view? Open the standalone lab. Your stars are saved in this browser and shown on the series hub.

What to do next

1. Measure today's validation failure rate per endpoint and split it into syntax, schema mismatch, prose and truncation.
2. Move structured endpoints to schema mode and confirm which JSON Schema features your engine version supports.
3. Make every field that can be absent nullable, and give enums an explicit unknown value.
4. Size max_tokens from the longest expected objects and alert on outputs that finish for length.
5. Cache compiled grammars and check whether compile time shows up in p95 time to first token.
6. Compare value distributions for each field before and after enabling constraints to spot invented values.

Sources and further reading

• Efficient Guided Generation for Large Language Models (Willard and Louf, 2023)
• XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models (Dong et al., 2024)
• SGLang: Efficient Execution of Structured Language Model Programs (Zheng et al.)
• Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models (Tam et al., 2024)
• JSON Schema specification
• vLLM documentation: engine arguments

Related in this series:

• Sampling Settings That Fix Real Failures: Temperature, Top-p and Min-p
• Logprobs and Calibration: Measuring When a Language Model Is Guessing
• Retrying Agent Tool Calls Without Executing the Same Action Twice

Diagrams, examples and the lab model are original educational material. Lab numbers are calculated by a synthetic model, not measured on real hardware or services.

Views are my own and do not represent any employer, past or present.