Why architecture matters here
Prompt pipeline architecture matters because prompts drift. An unversioned prompt gets edited by three different people over a month; you can't reproduce yesterday's behavior; a regression takes weeks to find. Version control, tests, and rollout discipline for prompts prevent this.
Cost matters. Every prompt decision — model choice, few-shot examples, output format, retry policy — has direct cost implications. A pipeline that picks the right model per intent runs 3-5x cheaper than one that sends everything to GPT-4. Explicit routing pays.
Quality is where the pipeline shines. Structured outputs (JSON schemas) plus retry-on-parse-failure turn hallucinations from a support problem into a caught-and-retried failure. Evaluation harnesses run before every prompt change, blocking regressions.
The architecture: every stage explained
Walk the diagram top to bottom.
User Input. The raw request. Chat message, form data, API call. Rarely useful directly; needs to be shaped into a prompt.
Prompt Template. A parameterized template with system prompt, user prompt, optional few-shot examples. Jinja2 is common; some teams use custom DSLs. Templates live in the registry.
Variable Store. The context needed to fill the template: user profile, session history, retrieved knowledge, tenant settings. Pulled from memory, session, or retrieval systems.
Prompt Compiler. Renders the template with variables. Validates: are all required vars present? Does the total token count fit the budget? Are dangerous patterns (prompt injection markers, PII) present? Runs a safety pre-check.
Structured Output Schema. A JSON schema or function definition the model must produce. Instructor, Pydantic-AI, Guardrails, and native provider function-calling all implement this. Structured outputs eliminate parsing errors as a class.
Model Router. Given the intent, picks the right model. Simple prompts → cheap fast model. Complex reasoning → larger model. Vision → multimodal model. The router is the biggest cost lever in the pipeline.
LLM Provider. OpenAI, Anthropic, Google, or a local model. Abstract the provider so switching is one config change. Retry logic handles transient failures; fallback logic handles sustained ones.
Validation. Parse the model output against the schema. If parse fails, retry with the error as feedback. If schema is valid, run business rules (allowed values, ranges, cross-field constraints). Reject invalid outputs before they hit downstream.
Prompt Registry. Prompts are code. Version them in git alongside the app. Each version has metadata: description, expected inputs, sample outputs, eval scores. Rollout follows the same discipline as code deploys.
Eval Harness. A golden set of inputs with expected outputs (or rubrics). Run on every prompt change; block promotion on regression. LangSmith, Braintrust, and Promptfoo are common tools; in-house harnesses work too.
Observability + A/B. Log every prompt and completion (with PII redaction). Track quality metrics per prompt version. Canary new versions to a small percentage before full rollout.
End-to-end request flow
Trace a request. A user asks the customer support bot "when will my order arrive?"
Prompt template "support_query" is loaded from the registry version pinned to the current release. Variables are populated: user profile, order history, session context. The prompt compiler renders the template, verifies token budget (below 3k), and passes it forward.
The output schema specifies { "answer": string, "confidence": "high"|"medium"|"low", "escalate": bool }. The model router recognizes this as a shipping query and picks GPT-4o-mini (fast, cheap, adequate).
The request goes to OpenAI. The model responds with structured JSON. Validation parses it, checks that confidence is one of the allowed values, verifies "escalate" is bool.
If parsing failed (very rare with structured outputs), the pipeline retries with the schema error as feedback. If succeeded, the pipeline returns the answer to the user, logs the prompt-completion pair for eval, and updates cost/quality metrics.
Meanwhile the eval harness has been running in the background on the canary prompt version. If it regresses on the golden set, the canary is aborted. If it improves, the new version rolls to more traffic.