AutoRAGBench, submitted November 19, 2026 at 06:26 UTC, evaluates nine agent configurations across nine retrieval tasks. Each starts without code and receives six hours and one H800 GPU.
Better scores do not remove the retrieval cost
All agent-built systems have higher mean retrieval latency than Reference RAG. Of 77 builds with comparable development scores, 20 finish at least one point below their peak. These are unreproduced author measurements.
Evaluation implication: preserve the selection rule as well as the final code. Choosing a checkpoint after seeing hidden-test results would answer a different question from asking whether an agent can select its own useful artifact.
A proposed corpus with confusing component names
Original exercise, not executed: create 60 short synthetic technical notes with deliberately similar component names. Provide development questions while keeping a disjoint set hidden. Compare a simple lexical retriever with the agent's changes using evidence correctness and retrieval latency.
Save candidate checkpoints and select one using development data alone. Then evaluate that frozen selection on the hidden questions, including cases where similarly named components require different evidence. Report latency and missed evidence separately rather than combining them into an unexplained score. This toy exercise would illustrate the evaluation discipline without reproducing AutoRAGBench's task suite or hardware budget.
Sources accessed November 21, 2026. Views are my own and do not represent any employer, past or present.
Sources and evidence
Evaluate the final retrieval artifact on hidden questions and latency, with checkpoint selection fixed before the hidden results are visible.