Three papers posted on November 10 look at signals teams rely on that measure less than they appear to.
Do agents follow their instruction files?
A study of agent context files, the project-level instruction files that coding agents read on every task, links real file snapshots to the commits agents produced and turns each actionable instruction into an executable check. The authors find "agents follow ACF guidance more often than not, but adherence splits sharply between instructions followed consistently and those never respected", with differences across instruction categories and agents. The abstract gives no overall rate, so the practical point is the split: some kinds of instruction work and some do not.
Saturation that lives in the measurement
A re-scoring of released runs on eight benchmarks, including SWE-bench Verified, Terminal-Bench, MLE-bench and ALE-Bench, argues that "much of this saturation lies in the measurement, not the tasks". Reducing each run to one pass/fail bit is too coarse: on SWE-bench Verified, neighbouring top-ten agents "differ by less than its tests' errors". It also caps scores at the reference: ALE-Bench's top ten tie when graded against the median human, "although nearly half the runs beat that human". In a Terminal-Bench re-run, models tied on pass/fail "differ over tenfold on the quantity the tests discard". The authors propose graded credit with cost reported as a separate axis.
Reusing a cache across model versions
A KV cache study asks whether caches built by one model version can serve the next, using DeepSeek-V4-Flash and an update called Flash-0731 among four updates. Reuse gave median time-to-first-token speedups of 2.10 to 5.08 times on three long-prefix benchmarks, but retrieval accuracy ranged "from near-native performance to large losses", and "even agreement on the first generated token does not ensure agreement in answer content or stopping behavior."
What remains uncertain. We read the abstracts only. The context-file study does not say which agents or instruction types fared worst in its abstract, the benchmark result depends on the authors' proposed scoring, and the cache study covers one model family in detail.
Analysis: At Belgavi's Book Store, Hansini could turn the most important lines of the help-desk agent's instruction file, such as never promising a refund date, into scripts that check every closed CASE, since the study shows some instructions are simply never followed. When comparing two agents, record how far each got, and at what cost, not only whether it passed. And after a model upgrade, flush cached context rather than keeping it for speed.
Sources and evidence
- Do They Do as They Are Told? Investigating Coding Agents' Adherence to Explicit Instructions in Context Files (arXiv, November 10)
- Agent Benchmarks Stop Ranking Before They Run Out of Tasks (arXiv, November 10)
- Can KV Cache Be Reused Across Model Updates? A DeepSeek-V4-Flash Case Study (arXiv, November 10)
Agent context-file instructions are followed more often than not, but some kinds are never respected, so check the important ones in code. Re-scoring eight agent benchmarks shows pass/fail grading hides large differences between tied models. Reusing KV caches across model updates speeds responses but can change answers. Measure compliance, grade by degree and flush caches after upgrades.