Research review / research

AutoRAGBench Tests Whether Agents Can Build Better Retrieval Systems

A November 19 preprint evaluates retrieval systems built by agents. Reported gains coexist with slower retrieval and development scores that sometimes decline before final submission.

By Sandeep Belgavi · Published 2026-11-20

AutoRAGBench, submitted November 19, 2026 at 06:26 UTC, evaluates nine agent configurations across nine retrieval tasks. Each starts without code and receives six hours and one H800 GPU.

Hidden evaluation examines the artifact the agent leaves behind

The preprint excludes test sets from development containers and evaluates evidence with five held-out answering models. Its highest average task score is 56.1 for GPT-6.1 Sol with Codex CLI, versus 44.4 for its composite Human-SOTA baseline. These average task-specific metrics, not pooled accuracy.

Better scores do not remove the retrieval cost

All agent-built systems have higher mean retrieval latency than Reference RAG. Of 77 builds with comparable development scores, 20 finish at least one point below their peak. These are unreproduced author measurements.

Evaluation implication: preserve the selection rule as well as the final code. Choosing a checkpoint after seeing hidden-test results would answer a different question from asking whether an agent can select its own useful artifact.

A proposed corpus with confusing component names

Original exercise, not executed: create 60 short synthetic technical notes with deliberately similar component names. Provide development questions while keeping a disjoint set hidden. Compare a simple lexical retriever with the agent's changes using evidence correctness and retrieval latency.

Save candidate checkpoints and select one using development data alone. Then evaluate that frozen selection on the hidden questions, including cases where similarly named components require different evidence. Report latency and missed evidence separately rather than combining them into an unexplained score. This toy exercise would illustrate the evaluation discipline without reproducing AutoRAGBench's task suite or hardware budget.

Sources accessed November 21, 2026. Views are my own and do not represent any employer, past or present.

Sources and evidence

Evaluate the final retrieval artifact on hidden questions and latency, with checkpoint selection fixed before the hidden results are visible.