Why architecture matters here
Prompt evaluation architecture matters because LLM behavior is non-deterministic and change is constant. A small prompt tweak can shift outputs materially. Eval is the discipline that catches this cheaply, before users do.
Cost is real. Each eval run costs LLM calls proportional to golden set size. Golden sets in the hundreds are typical.
Reliability comes from a golden set that reflects reality. Bad golden set = confident regression on a case that doesn't matter, or blind to one that does.
The architecture: every piece explained
Walk the diagram top to bottom.
Prompt version. The candidate under test. Semantic version + git ref.
Golden Set. Curated inputs + expected outputs (or rubrics). Cover happy path, edge cases, adversarial.
Runner. Executes the candidate against each golden input; collects outputs, latency, cost.
Metrics. Deterministic (exact match, regex, JSON schema conformance), LLM-judge (rubric-based), human review sample.
Regression Gate. Compare to baseline. Fail if any metric drops beyond threshold.
CI Integration. On every PR, run eval; block merge on regression. Fast enough (parallel runs) to be practical.
Cost + Latency. Budget checks: candidate must not blow up cost/token or p99 latency.
Adversarial Tests. Jailbreak attempts, PII, ambiguous phrasings. Regression on these matters extra.
Rollout Gates. Even after CI passes, canary in prod; A/B against baseline; watch metrics; ramp.
Continuous Curation. New failures → new golden cases. Set grows over time.
End-to-end evaluation flow
Trace a change. Developer edits a support prompt to be more concise. Opens PR.
CI runs eval on golden set of 400 support queries. For each, runs the new prompt, collects response.
Metrics: exact-match on structured outputs (100% pass); LLM-judge scores concise-ness (up), helpfulness (down slightly), factual (unchanged).
Regression gate: helpfulness dropped 0.3 pts, threshold is 0.5. Passes.
Cost + latency: cost per query down 15%; latency down 8%. Good.
Adversarial: no regression on jailbreak resistance.
CI green. PR mergeable.
Rollout: canary 5% traffic. Real-user metrics track. Two days in, CSAT stable, escalation rate marginally down. Ramp to 25%, 100%.
Two weeks later, a novel query type causes bad outputs. Added to golden set. Future prompt changes must handle it.