Why architecture matters here
The architectural significance of structured output is that it converts LLM integration from parsing to typing. Unconstrained, the contract between model and consumer is a hope enforced by regex and try/except; every downstream component inherits the tail risk, and the failure modes are the worst kind — rare, input-dependent, and discovered in production. Constrained, the contract is a schema the decoder physically cannot violate: the JSON parses, the enum is one of the enumerated values, the required fields exist. Whole categories of defensive code — repair heuristics, retry-on-parse-failure loops, 'strip everything before the first brace' — become unnecessary, and reliability stops being proportional to prompt-engineering diligence.
It matters equally for what it enables upstream: agents and tools. Function calling is structured output wearing its production clothes — the model selects a tool and emits arguments conforming to the tool's parameter schema. Every agent framework's tool loop rests on that guarantee; argument hallucination (a string where an integer belongs, an unknown enum) is exactly what constrained decoding eliminates, which is why schema quality has become part of tool design. The same mechanism powers extraction pipelines (documents → typed records at scale), classification with guaranteed label sets, and UI generation where malformed output would crash a renderer.
And it matters because the constraint is not free — knowing the physics keeps you out of two ditches. Compute-wise, mask computation must keep pace with decoding (microseconds per step via precompiled automata, or it becomes the bottleneck at batch size 200). Quality-wise, constraints interact with reasoning: force a model to emit a bare label immediately and you amputate its chain-of-thought; the standard remedy — a free-text reasoning field before the answer field, or think-then-constrain two-phase generation — is an architectural pattern, not a prompt trick. Teams that treat structured output as 'turn on JSON mode' get correctness with silent quality loss; teams that design the schema as part of the cognition get both.
The architecture: every piece explained
Top row: from schema to sampler. The schema enters as JSON Schema (typically via Pydantic/dataclass definitions in application code) or, for non-JSON structures, a context-free grammar (GBNF, Lark). Grammar compilation turns it into a decision structure: regex-convertible schemas become finite-state machines; recursive structures (nested objects, arrays) get pushdown handling. The output is, conceptually, a map from automaton state to the set of legal next characters — which must then be lifted to token masks: for each state, which of the model's ~100k vocabulary tokens (each a multi-character chunk) keep the automaton on a legal path. Precomputing that state→mask index is the trick that makes per-step overhead microseconds (Outlines' FSM indexing, llguidance's lazy computation); naive per-step scanning of the vocabulary is the version that melts throughput. The decoder loop then does the simple thing: logits += mask (−∞ on illegal tokens), sample, advance the automaton, repeat — plus subtleties like forcing EOS when the structure completes and handling max-token truncation mid-structure.
Middle row: where it runs. Tokenizer quirks are the perpetual tax: tokens spanning structural boundaries (",\n " as one token), multiple tokenizations of the same string, and whitespace variants all must be handled by the lifting, not banned outright — over-restrictive masks measurably hurt output quality by forbidding the token sequences the model prefers. Engine support: vLLM and TGI expose guided_json/guided_grammar (with backends like Outlines, llguidance, XGrammar); llama.cpp takes GBNF grammars; SGLang integrates constraints with its programming model. API-level modes on hosted models: JSON mode (valid JSON, no particular schema), response schemas (conform to this structure — the strong form), and function calling, where the API constrains tool-argument generation against declared parameter schemas and returns typed calls.
Bottom rows: the pragmatic wrapper. A validation layer persists even with constraints: semantic validation (the date parses but is in the past; the SKU exists), truncation detection, and — for API modes weaker than full constraint — parse-retry-repair fallbacks. Quality effects get measured, not assumed: reasoning-field-first schemas, field ordering that matches natural generation order, and enums over free text where the label set is closed. The ops strip: schemas versioned like APIs (they are), rejection/retry metrics as the canary for schema-model mismatch, and per-request latency overhead tracked against the unconstrained baseline.
End-to-end flow
Follow an extraction pipeline in production. The task: turn supplier invoices (PDF → text) into typed records — vendor, line items (description, quantity, unit price), currency, total, payment terms. The schema is a Pydantic model: Invoice with a line_items: list[LineItem], currency as a 3-letter enum, amounts as decimals-as-strings (a deliberate choice — float tokens invite drift), and one deliberate extra: reasoning: str as the first field, where the model works through ambiguities before committing values.
Serving runs on vLLM with guided JSON. At startup, the schema compiles once to an FSM index (a few hundred milliseconds, cached by schema hash). At inference, a batch of 64 invoice extractions decodes concurrently; each sequence carries its automaton state; the mask lookup per step is a table hit, adding ~2% latency versus unconstrained. Every response parses — the pipeline's try/except JSON block has fired zero times in six months. What still fires is the semantic validator: line-item totals that don't sum to the stated total (the model mis-read a column) route to a second-pass prompt with the discrepancy quoted, and 0.3% of documents land in the human queue with the reasoning field attached — which, the reviewers report, is the most useful debugging artifact in the product.
The same architecture serves the team's agent. Tools are declared with parameter schemas; the model's tool calls are constrained to them, so issue_refund(amount="full") — the string-where-number bug that once crashed the handler — cannot be generated; the amount field admits only digits. When the team A/B-tested constraint effects, the measurable finding matched the literature: on classification-style outputs, immediate-answer schemas cost ~4 points of accuracy versus reasoning-first schemas — the constraint wasn't the problem; the amputated thinking was. Their standing rule now: every constrained schema gets a scratchpad field unless latency forbids it, and schema changes ship through the same eval gate as prompt changes, because a schema is a prompt — the strongest one in the request.