Why architecture matters here

Parsing fails on hallucinated fields, malformed JSON, and no retry. Architecture matters because constrained decode + parser + retry combine.

Advertisement

The architecture: every piece explained

The top strip is the primary path. Prompt with schema. JSON mode / grammar constrained decode. Raw output. Parser strict + lenient.

The middle row is recovery. Repair pass minor fixes. Retry with feedback. Validation schema. Fallback smaller model or default.

The lower rows are practice. Structured tools function calling. Metrics parse success. Ops — schema versioning + eval + guardrails.

Output parsing — JSON mode + grammars + retry + repairextract structured data from LLM outputPrompt with schemainstruction + exampleJSON mode / grammarconstrained decodeRaw outputpossibly imperfectParserstrict + lenientRepair passfix minor issuesRetry with feedbackon parse failValidationschema checkFallbacksmaller model or defaultStructured toolsfunction callingMetricsparse success rateOps — schema versioning + eval + guardrailsrepairloopvalidatefallbackprefermeasuremeasureoperateoperate
Output parsing pipeline with repair + retry.
Advertisement

End-to-end flow

End-to-end: prompt asks for JSON matching schema. JSON mode on. Output parses. Validation passes. If parse fails, repair fixes trailing comma; if that fails, retry with parser error as feedback; if that fails, fallback to smaller model. Success rate 99.6%.