Sample prod traffic
1-5% of turns get judged in real time. Judge scores flow to dashboards.
Advertisement
Watch for drift
Judge pass rate week-over-week. Sudden drop = something changed (model, retrieval, prompt, upstream).
Advertisement
Alert on quality drops
Quality is a metric. Alert like you would on latency drops.