Why architecture matters here
Agent eval fails without offline + online + trace comparison. Architecture matters because gates + replay + judge together turn eval into a reliable practice.
The architecture: every piece explained
The top strip is the pipeline. Task suite product-representative. Offline eval batch. Online eval A/B. Trace comparison candidate vs prod.
The middle row is quality. Judge model scores. Golden set must not regress. Metrics quality + latency + cost. Regression gate in CI.
The lower rows are ops. Trace replay deterministic. User feedback loops. Ops — governance + versioning + audit.
End-to-end flow
End-to-end: PR updates agent. Offline eval runs 500 tasks; judge scores. Golden set unchanged. Latency + cost measured. Regression gate in CI passes. Deployed as 5% shadow. Online eval compares trace outcomes. Wins on quality; 100% rollout.