Why architecture matters here
Structured output architecture matters because LLM prose is unreliable to parse. A model that returns "Sure, the answer is yes" needs regex; another day it says "Certainly, yes it is." Structured output makes this deterministic: {"answer": true}. The class of bugs disappears.
Cost is modest and often net-negative. Structured output may cost a fraction more per call (function calling adds tokens for schema) but eliminates retries from parse failures, which is a bigger cost.
Reliability jumps. Retry-on-error patterns can push parse success rates from 95% (best-effort prose) to 99.9%+ (structured with retries). At scale this is the difference between "sometimes fails" and "reliable production."
The architecture: every piece explained
Walk the diagram top to bottom.
Model + prompt. Standard LLM call with system + user messages. System prompt says the response format is required.
Schema. JSON Schema or a Pydantic model. Defines fields, types, required, and constraints (enum, min/max).
Provider Feature. OpenAI's function calling / structured outputs, Anthropic tool use with schema, Gemini function calling. Provider constrains the model to produce valid JSON per the schema.
Constrained Decoding. At token level, the sampler restricts to tokens that continue a valid JSON matching the schema. Guarantees valid output but subtly affects distribution.
Post-parse Validation. After receipt, parse and validate against the schema. Even with provider constraints, semantic constraints (business rules, cross-field) may fail. Retry with the error as feedback.
Instructor / Guardrails / Outlines. Libraries that abstract provider differences. Instructor uses Pydantic + retries. Guardrails supports validation and re-ask. Outlines does grammar-guided decoding.
Retry with error. If parse or validation fails, resend the prompt with the specific error and instructions to fix. Usually succeeds on retry.
Fallback. If retries exhaust, degrade gracefully — simpler schema, freeform response with wrapper, or human escalation.
Streaming Structured Output. Some providers stream partial JSON. Client can render as fields fill.
Observability. Parse success rate, retry rate, per-schema cost. Regression detection.
End-to-end structured output flow
Trace a call. You have a support-classification task. Schema: {ticket_id: str, category: enum, urgency: 1-5, needs_human: bool, summary: str}.
Using Instructor with OpenAI: define Pydantic model TicketClassification with those fields. Call client.chat.completions.create(response_model=TicketClassification, messages=[...]).
Instructor sends the schema as a function-call. Provider constrains decoding. Model returns valid JSON.
Instructor parses into TicketClassification. Pydantic validates types and constraints. All pass; return the object.
Retry scenario: model returns urgency=7 (outside range). Pydantic raises ValidationError. Instructor sends a re-ask with the specific error: "urgency must be between 1 and 5." Model corrects; second attempt validates.
Streaming scenario: use response_model with streaming. As tokens arrive, Instructor incrementally parses; user sees the ticket_id appear first, then category fills in, etc.
Observability: log parse_success_rate = 99.4% over the last hour; retry_rate = 3%. Alert if retry rate crosses 10% — probably a schema or prompt bug.