Research review / trendwatch

Which Harness Fits Your Model? A New Benchmark Fuels Debate

Runta's November 17 FrontierHarness evaluation compares 22 configurations across five models. Its Hacker News discussion questions task selection and the meaning of spending on failures, making this a concrete debate about how to choose an agent harness.

By Sandeep Belgavi · Published 2026-11-18

Runta published FrontierHarness Eval v1.1 on November 17, 2026, in collaboration with Baseten. It reports 27,280 trials across 22 harness configurations, five models and 62 tasks, with four attempts per combination. The study concerns these particular model-and-runtime combinations, not every coding agent.

A visible dispute about the comparison

The November 17 Hacker News archive includes the study. Its discussion showed 338 points and 118 comments when checked November 18. Participants challenged why the 62 tasks were chosen and whether spending on failed trials should be described as waste. A poster explaining the study supplied runtime and scoring details. This establishes a dated debate on one forum, not developer consensus.

Read the cost denominator

Runta estimates inference costs using token counts and list prices. It reports different best-value configurations for different models and says 64.5% of scored-run cost went to unsuccessful trials. Its scope is five open models served by one provider. The authors also report that several apparent winners are close under task resampling.

A failed attempt can still generate information, but that does not make its cost disappear. For a deployment decision, define what counts as an accepted result, include retries and checking, and keep the task distribution visible. Neither a low price per attempt nor a high pass rate alone gives the complete comparison.

A small pairing experiment

Original proposed evaluation, not executed: choose two harnesses and two model versions for the same synthetic repository tasks. Pin tool permissions and stopping budgets. Run repeated trials with an independent verifier, then report accepted outcomes, elapsed time and total cost for all four pairs.

Keep failed and timed-out attempts in the accounting, with infrastructure failures identified separately. Publish the selection rule before examining the results. This would make a local choice reviewable without claiming to reproduce the larger benchmark.

I would use this debate to improve a comparison protocol before changing a harness. The reported variation is a reason to test pairings, while the questions about task selection and close winners are reasons to resist a universal ranking. For my own synthetic tasks, I would choose an acceptance threshold first, keep unsuccessful runs in the cost ledger and change the model and harness separately. That would show whether an apparent saving survives the work I actually need completed.

Sources accessed November 18, 2026. Views are my own and do not represent any employer, past or present.

Sources and evidence

Compare model-and-harness pairs on a declared task set, with repeated trials and the cost of unsuccessful attempts included.