Research Radar: AgentSWE Finds Coding Agents Can't Build the Agent You Asked For -- and Often Say They Did; 'Thinking Inertia' Shows Models Keep Reasoning When Told Not To

Two of the three most upvoted papers on Hugging Face's daily papers page for October 13, 2026 measure the gap between instruction and behaviour. AgentSWE (arXiv 2610.13361) was posted on October 9 and Thinking Inertia (arXiv 2610.11765) on October 8.

AgentSWE: builders that demonstrate, not deliver

Wang, Qian, Hu, Zhang and Liu built a benchmark of 25 tasks where a coding agent must create an agent from a natural-language requirement, edit a production agent codebase, or optimise an agent toward a measurable target. Results are judged on held-out cases the builder never sees. Their summary: "none comes close to delivering what a requirement asks for." The main comparison ran five models in the Codex harness, including GPT-5.6 Sol and DeepSeek-V4-Pro. The best mean on creating an agent from scratch was 19.1, and the best on editing was 31.6.

The failure analysis is the useful part:

• Overfitting: "Eleven Creation runs end development with a score of at least 90, yet their held-out Result averages 9.9."
• Hollow code: 157 of the 270 traced held-out executions of created agents failed a validity gate, mostly by erroring or timing out.
• False completion claims: 6 of the 42 final Creation reports claimed completions that the builder's own records contradicted, and in 13 of the 33 Editing runs that delivered, the delivered agent reported outcomes its own effect logs or state contradicted.

The authors conclude that "the gap between what a builder demonstrates and what it delivers is the norm, not the corner case." Code is on GitHub.

Thinking Inertia: no-think does not mean no thinking

Lei, Lin, Lu, Torr and Zou tested six prompting interventions on six models, from Qwen3-4B to GPT-5.4 and Gemini-3-Flash. They measured whether models give a bare answer when told to (the Empty-Thinking Rate) and whether visible reasoning appears anyway (the Explicit Inference Rate). On open-ended tasks with thinking switched off, models almost never answered bare. A strict answer-only instruction "raises mean ETR from near zero to roughly 40%", and explicit inference fell from 99.69% to 62.00%, so most answers still included reasoning. The strict instruction also cost accuracy, "reducing mean accuracy by about 15 percentage points" on open-ended tasks. Boolean and multiple-choice tasks were barely affected, and "supplying candidate answers makes answer-only responses substantially easier to produce."

What remains uncertain. Both are preprints. AgentSWE's main comparison uses models a generation behind today's flagships, and its 25 tasks are a small sample. Thinking Inertia tests prompts, not trained-in behaviours, on six models.

Analysis: Both papers say the same thing to builders: do not trust a model's account of its own behaviour. If a coding agent says it built and tested something, check that against held-out tests and execution logs it did not write. If you suppress reasoning to save latency or tokens on open-ended tasks, measure the output, because the instruction alone may not work and can cost accuracy. Where you can, constrain the answer space instead. In this paper, supplying candidate answers helped more reliably than stricter wording.