Most agent evaluation asks one question: did the task end in the right state? Several items this week show that a correct-looking end state can hide a wrong route, and that the route is where the risk lies. This differs from the proof-carrying work we described on November 9. A proof checks a claim, and these failures happened in work that appeared to pass.
Five signals
- The wrong bug. OpenAI's November 4 Linux kernel report, whose headline count of 948 root exploits we covered on November 6, says 20 of 200 early "successful" exploits "achieved root privileges through a different 0-day vulnerability than the one assigned". The fix was an independent check that each exploit used its assigned bug.
- Answers no agent wrote. A study of 95 failed multi-agent runs found that 45 percent of one corpus's hand-crafted failures ended with a content-filter crash, "after which the harness emitted a final answer no agent produced".
- Evidence already present. The same study found evidence against the error inside the trajectory in 72 percent of failed runs, and in 88 percent of those no agent acted on it.
- Skills that still look fine. StealthyPoison changes a skill's decision rules "while preserving its advertised functionality and normal execution", so a skill that runs without errors proves little.
- Labels taken at their word. A Codex change lets an MCP tool's read-only hint skip approval in automatic mode, while the MCP specification calls such hints untrusted unless the server is.
What checking the route looks like
The fixes in these cases share a pattern. They record which input, rule or vulnerability the agent actually used, and compare it against what it was assigned. They keep the check outside the agent's control, as OpenAI did with its independent exploit check. And they treat a final answer as valid only if it can be traced to a step an agent took. None of this needs a new model. It needs the harness to keep and compare provenance it often throws away.
What remains uncertain. These are a few examples selected because they share a pattern. Route checks add cost and latency, and we have not seen measurements of how much they catch in production compared with checks on outcomes alone.
Analysis: At Belgavi's Book Store, a refund that matches the order total can still be wrong if the agent applied the wrong return policy to reach it. Geetha's team could log which policy rule and which order record each refund decision used, and have a separate script confirm that both match the CASE before the refund goes out. Start with the decisions that cost money, and add route checks to an evaluation suite wherever a test currently checks only the final state.
Sources and evidence
- A thousand fewer paths to root (OpenAI, November 4)
- Dependable Long-Horizon LLM Agents: A Trajectory-Centered Empirical Study of Where Errors Enter and Which Controls Reach Them (arXiv, November 9)
- StealthyPoison: Manipulating LLM Agent Behavior through Procedural Poisoning of Third-Party Skills (arXiv, November 9)
- Honor read-only MCP tool hints in automatic approval (Codex #60638)
- Tools: annotations (Model Context Protocol specification, 2025-06-18)
This week's evidence shows agents reaching correct-looking end states by wrong routes: exploits using the wrong bug, harnesses reporting answers no agent wrote, and poisoned skills that still run normally. Checking only the outcome misses these failures. Record which inputs and rules an agent used, compare them outside the agent's control, and start with decisions that cost money.