Brian T. Weston, Alondra Torres Contreras and Aldair E. Gongora submitted SciResearch-Eval on November 16, 2026; it appeared in arXiv's November 17 listing. Agents infer thermal properties through experiments with potentially flawed simulators.
Confidence is part of the answer
Across 93 held-out tasks and nine models, the authors report 20-62% coverage for most models' stated 95% intervals when physics is missing, versus 94% for GPT-6 Astra and 99% for Claude Fable 5.2 at default settings. These are unreproduced study results, not general capability rankings.
The paper scores both coverage and interval quality. Asking for wider ranges did not solve the problem. For a numerical workflow, the distinction matters: a successful tool call and a convincing fit can coexist with a wrong physical assumption.
Experimental design can avoid a simulator's weak region
Stronger agents sometimes chose gentle heating that suited the simplified simulator. The implication is to evaluate how an agent obtains evidence, alongside how it interprets the returned values. A generic technical analogue is selecting inputs that distinguish two competing explanations instead of repeatedly fitting the same convenient cases.
Limits and an original transfer exercise
Limitations include one physical domain, no human baseline, unequal reasoning budgets and a harness difference for Fable's later runs. These qualifications matter before transferring coverage rates to another workflow.
Proposed synthetic exercise, not executed: generate observations from a quadratic curve but provide an agent with only a linear fitting tool. Let it choose sample locations, inspect residuals and return a prediction interval for a withheld input. Score the interval against the known synthetic generator and record whether the agent requests evidence outside the initially convenient region.
This is an original toy exercise about model mismatch, not a reproduction of SciResearch-Eval. The authors release their environment and harness under CC BY 4.0; the linked paper and project provide the required attribution. No benchmark code or dataset is copied here.
Sources accessed November 18, 2026. Views are my own and do not represent any employer, past or present.
Sources and evidence
Test whether an experimental agent changes its evidence-gathering strategy when the supplied simulator is wrong, and score uncertainty alongside point estimates.