Prompt Pipeline Architecture in Depth

The request path, stage by stage

Template. A template is a parameterised system message, user message and optional few-shot examples. Keep it in a file or registry, never inline in business code, and give it an identifier and a version. Jinja2 or plain str.format is enough. The important property is that the template declares its variables, so a missing one fails loudly instead of rendering an empty string.

Variable store and context assembly. This stage gathers what the template needs: the user's profile and tenant settings, recent conversation turns, retrieved documents, tool results. Retrieval is usually the largest and least predictable input, so assemble context in priority order and know in advance what gets dropped first when space runs out.

Compiler. The compiler renders the template, counts tokens and enforces a budget. It also runs cheap checks before any money is spent: required variables present, no secrets or unredacted PII in the rendered text, untrusted content fenced and labelled as data rather than instructions. Rendering is deterministic, so the compiled prompt can be hashed and logged, and that hash is what makes a bad answer reproducible later.

Router. The router chooses a model and settings for this request. Common inputs are task type, input length, the tenant's tier and the current health of each provider. It is the biggest cost lever in the pipeline, because most traffic in most products is easy, but it adds a decision that can be wrong. Evaluate it like any other component.

Provider call. Wrap each provider behind one interface with a timeout, bounded retries with jittered backoff on rate-limit and server errors, and a fallback model for sustained outages. Retries must be safe. If the model can call tools with side effects, make those tools idempotent or the retry will act twice.

Validation. Parse the output against a schema, then apply business rules the schema cannot express. On failure, try one repair round that sends the validation error back to the model, and after that return a safe default or escalate. Never pass unvalidated text to code that acts on it.

The request path, stage by stage

Template. A template is a parameterised system message, user message and optional few-shot examples. Keep it in a file or registry, never inline in business code, and give it an identifier and a version. Jinja2 or plain str.format is enough. The important property is that the template declares its variables, so a missing one fails loudly instead of rendering an empty string.

Variable store and context assembly. This stage gathers what the template needs: the user's profile and tenant settings, recent conversation turns, retrieved documents, tool results. Retrieval is usually the largest and least predictable input, so assemble context in priority order and know in advance what gets dropped first when space runs out.

Compiler. The compiler renders the template, counts tokens and enforces a budget. It also runs cheap checks before any money is spent: required variables present, no secrets or unredacted PII in the rendered text, untrusted content fenced and labelled as data rather than instructions. Rendering is deterministic, so the compiled prompt can be hashed and logged, and that hash is what makes a bad answer reproducible later.

Router. The router chooses a model and settings for this request. Common inputs are task type, input length, the tenant's tier and the current health of each provider. It is the biggest cost lever in the pipeline, because most traffic in most products is easy, but it adds a decision that can be wrong. Evaluate it like any other component.

Provider call. Wrap each provider behind one interface with a timeout, bounded retries with jittered backoff on rate-limit and server errors, and a fallback model for sustained outages. Retries must be safe. If the model can call tools with side effects, make those tools idempotent or the retry will act twice.

Validation. Parse the output against a schema, then apply business rules the schema cannot express. On failure, try one repair round that sends the validation error back to the model, and after that return a safe default or escalate. Never pass unvalidated text to code that acts on it.

A minimal pipeline in code

The sketch below is provider-neutral. call_model stands for whatever SDK you use; everything else is standard Python with Pydantic for the schema.

import hashlib, json, string, time
from dataclasses import dataclass
from pydantic import BaseModel, ValidationError, field_validator

class Triage(BaseModel):
    category: str            # one of CATEGORIES
    priority: int            # 1 (urgent) .. 4 (low)
    summary: str

    @field_validator("priority")
    @classmethod
    def in_range(cls, v):
        if not 1 <= v <= 4:
            raise ValueError("priority must be 1..4")
        return v

@dataclass(frozen=True)
class PromptVersion:
    prompt_id: str; version: str; system: str; user: str; max_input_tokens: int

def compile_prompt(pv: PromptVersion, vars: dict, count_tokens) -> tuple[list, str]:
    required = {f for _, f, _, _ in string.Formatter().parse(pv.user) if f}
    missing = sorted(required - vars.keys())
    if missing:
        raise KeyError(f"missing vars: {missing}")
    msgs = [{"role": "system", "content": pv.system},
            {"role": "user", "content": pv.user.format(**vars)}]
    if count_tokens(msgs) > pv.max_input_tokens:
        raise ValueError("over budget: trim context before calling the model")
    digest = hashlib.sha256(json.dumps(msgs, sort_keys=True).encode()).hexdigest()[:16]
    return msgs, digest

def run(pv, vars, route, call_model, count_tokens, log):
    msgs, digest = compile_prompt(pv, vars, count_tokens)
    model = route(vars)
    for attempt in range(2):                     # one original call + one repair round
        t0 = time.monotonic()
        raw = call_model(model, msgs, timeout_s=20)
        try:
            out = Triage.model_validate_json(raw)
            if out.category not in vars["categories"]:
                raise ValueError(f"unknown category {out.category!r}")
            log(prompt=pv.prompt_id, version=pv.version, digest=digest, model=model,
                attempt=attempt, ok=True, ms=int((time.monotonic() - t0) * 1000))
            return out
        except (ValidationError, ValueError) as e:
            log(prompt=pv.prompt_id, version=pv.version, digest=digest, model=model,
                attempt=attempt, ok=False, error=str(e)[:200])
            msgs = msgs + [{"role": "assistant", "content": raw},
                           {"role": "user", "content": f"That output was invalid: {e}. Return corrected JSON only."}]
    return Triage(category="needs_human", priority=2, summary="automatic triage failed")

Three details carry most of the value. First, the prompt version and the digest of the compiled messages are logged with every call, so any output can be traced to the exact text that produced it. Second, there is exactly one repair round, so a stubborn failure costs at most two calls rather than looping. Third, the fallback is a typed, explicit value that downstream code already knows how to handle.

What structured output does and does not guarantee

Structured output features come in two strengths. Some providers offer constrained decoding against a JSON schema, where the sampler can only emit tokens that keep the output valid. That removes malformed JSON within the subset of schema features the provider supports. Others, and every provider in plain JSON mode, only make valid output much more likely, so parse failures become rare but not impossible. Check your provider's documentation for which applies and which schema keywords are honoured.

Neither kind guarantees that the content is right. A perfectly valid object can carry an invented category, a priority that contradicts the summary, or a refusal squeezed into a string field. That is why the pipeline has a second layer of business-rule validation and why evals score the content rather than the format. Also budget for truncation. If generation hits the output token limit, even constrained output can stop mid-object, so check the finish reason before parsing. Libraries such as Instructor, Pydantic AI and Guardrails package the parse, validate and re-ask loop; structured output covers the trade-offs.

The release path: registry, eval gate, canary

A prompt change is a behaviour change, so it ships like code. The release path has four parts.

1. Registry. Every version is immutable and carries its template, variables, target model, sampling settings and a changelog note. Production pins a version explicitly. The label latest is for development only.
2. Golden set. A few hundred real inputs, de-identified, with expected outputs or rubric scores and a slice tag for each, such as language, product area or customer tier. Add every production failure you fix as a new case.
3. Eval gate. Run candidate and baseline on the same set with the same model and settings. Promote only if the overall score is not worse and no important slice regresses beyond a set tolerance. Repeat noisy cases, because sampling variance can make a small change look like a regression or an improvement.
4. Canary. Route a small share of live traffic to the candidate, compare online metrics such as validation failure rate, escalation rate, user corrections, latency and cost, then promote or roll back by changing the pinned version. Rollback must be a configuration change, not a deploy.

Model upgrades go through the same gate. A new model version is a change to every prompt that uses it, and prompts tuned to one model's habits often lose a few points on another. Prompt versioning covers naming schemes and storage.

Worked example: shipping a triage prompt change

A support team runs support_triage v14, which assigns one of nine categories and a priority to incoming tickets. Agents complain that billing disputes are being filed as general questions. A prompt engineer writes v15, which adds two few-shot examples of disputes and a sentence defining a dispute.

The golden set has 420 cases across nine category slices. Results, three runs each at the production temperature, averaged:

Overall accuracy improved, and a gate that looked only at the headline number would have shipped v15. The slice rule blocked it. Refunds fell by almost eight points because the new examples taught the model that any mention of a charge suggests a dispute. The engineer added a contrasting refund example and a one-line boundary rule, giving v16: overall 93.6 percent, disputes 91.7 percent, refunds 94.1 percent. v16 passed the gate.

v16 then ran as a 5 percent canary for two days. Validation failures were unchanged, the rate at which agents re-categorised tickets fell from 7.9 to 5.2 percent, and cost per call rose about 19 percent because of the longer prompt. The team accepted the cost, promoted v16 by changing the pinned version, and added new refund-versus-dispute boundary cases to the golden set. Total engineering time was about a day, most of it spent reading failures rather than writing prompt text.

Observability: what to log on every call

Log one structured record per model call: prompt id and version, compiled-prompt digest, model, input and output tokens, latency, finish reason, validation outcome, attempt number and the caller's request id. Store full prompt and completion text only after redaction and under a retention policy, because those logs will contain customer data. From those records you can build the dashboards that matter: validation failure rate per version, cost per successful result rather than per call, latency percentiles per model, and fallback rate per provider.

Two signals catch regressions early. One is a jump in repair-round frequency for a version, which means the model and the schema disagree more than before. The other is a drift in the output category mix, which often means the inputs changed rather than the prompt, for instance after a product launch.

Failure modes and trade-offs

The main trade-off is ceremony against speed. A registry, gate and canary add a day to a prompt change. For an internal tool with one user, a git-tracked template and a 30-case smoke test are enough. For anything that acts on customer data or money, the full path pays for itself the first time it blocks a regression like the one above.

What to do next

1. Move every inline prompt into versioned templates with declared variables, and log the version on each call.
2. Add a compiler step that renders, counts tokens against a budget, and hashes the result.
3. Define a schema for every output that code consumes, add business-rule validation, and cap repair at one round.
4. Build a golden set of at least 200 real, de-identified cases with slice tags, and run it on every prompt or model change.
5. Make promotion and rollback a pinned-version config change, and canary new versions on a small traffic share.
6. Dashboard validation failures, repair rate and cost per successful result per version.
7. Read prompt routing before adding a second model, and evaluate the router itself.