Research Radar: Doc2Harness Finds the Best Model Builds a Working Agent Harness on Only 53% of Tests; a 12,000-GPU-Hour Study Finds No Self-Distillation Method Reliably Beats GRPO

Doc2Harness: can a model build the harness?

Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents (arXiv:2610.20344, submitted October 14, 2026) is by Yuyao Ge, Shenghua Liu and colleagues, with Jiafeng Guo and Xueqi Cheng among the authors. Its starting point: an agent's performance "depends as much on the harness around the model as on the model itself", yet existing benchmarks score tasks "under a fixed harness, leaving harness engineering itself unexplored."

The setup has two stages. A build model receives "an interface-only contract skeleton and a development document that consolidates behavior across twelve frontier harnesses", and builds a complete harness in an isolated container. That frozen harness is then run end to end by a scripted mock model, one container per test case. Each test is scored on four observable channels: "the requests sent to the model, the files on disk, the process exit code, and the command output". Positive controls rule out tests that pass by doing nothing.

The result: "Across nineteen frontier models, each evaluated with three independent builds on 328 test cases, GPT-6 Astra achieves the highest pass rate at 53.05%, while the weakest model reaches 27.44%". Even the strongest model "fails nearly half of the test cases, and failures cluster around how cross-cutting functionality is split across modules."

Self-distillation: a reality check

More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning (arXiv:2610.19789, also October 14) is by Linhao Zhong, Bozhen Fang and colleagues, with Chunhua Shen as senior author. It targets a habit in the field: evaluations that "reuse published baseline scores under differing experimental conditions". The authors compare ten self-distillation methods against plain GRPO on ALFWorld, ScienceWorld, Search-QA and WebShop at three model sizes, in "132 training runs totaling 12,124.4 recorded H800 GPU-hours", with eight rollout seeds for evaluation.

The three findings are blunt. "First, gains over GRPO remain limited in scope, with no method consistently outperforming it across settings. Second, strong final performance can mask unstable training. Third, additional training cost does not consistently translate into better performance."

What remains uncertain. Both are new preprints. Doc2Harness's specification is built from twelve existing harnesses, so it measures how well models reproduce current designs, not invent better ones. The self-distillation study covers four environments and three sizes, and methods may behave differently at frontier scale.

Analysis: Both papers point at the part of agent work that is easy to overstate. Harness code, meaning routing, tool plumbing, state and logging, is where cross-cutting bugs live, and the best model still fails nearly half of these tests. Teams letting agents modify their own scaffolding should test the scaffolding on what it actually sends to the model and writes to disk, as Doc2Harness does. Training teams have a cheaper lesson. Before adopting a new agent-RL method, reproduce the GRPO baseline under the same conditions, and track stability and cost alongside the final score.