Why architecture matters here
Prompt evaluations fail when they don't reflect production. Curated test cases become stale; judge models drift; human review lags. The architecture matters because the eval pipeline must keep pace with product changes.
With the pieces mapped, prompt eval becomes a durable practice.
The architecture: every piece explained
The top strip is the pipeline. Test case bank holds input + expected. Rubric defines criteria + weights. Judge model scores outputs against rubric. Regression runner executes on every change.
The middle row is quality guards. Metrics track accuracy + latency + cost. Golden set is a subset that must never regress. Human review samples judge decisions. Live shadow compares candidate against prod.
The lower rows are ops. Versioning keeps prompts in git. CI integration auto-runs eval on PR. Ops handles reproducibility, drift, and user feedback.
End-to-end flow
End-to-end: a team updates a support prompt. PR triggers regression: run 500 test cases through candidate; judge model scores; report diffs. Golden set of 50 cases must all pass. Human reviewer samples 10 judge decisions to calibrate. Live shadow runs candidate on 1% of prod traffic; compares outcomes. Merge only when all green. Users see stable quality.