Research review / researchradar

Research Radar: A Self-Improving Agent Can Delete a Safeguard Without Its Acceptance Gate Noticing

Papers submitted to arXiv on October 28 find that acceptance gates let overlapping safeguards erode unseen, that injected runtime feedback steered commercial coding agents in about 60 percent of trials, and that malicious CLAUDE.md files had no consistent effect on code security. Two more look at agent-written papers and agent repairs.

By Sandeep Belgavi · Published 2026-10-30

These preprints were submitted on October 28, 2026. All results are the authors' own and have not been peer reviewed.

Masked erosion of redundant safeguards

Self-improving agents edit their own harness, the prompts and rules around a fixed model, and keep an edit if an acceptance gate scores it higher on a batch of tasks (arXiv 2610.38392). The authors point out that when safeguards overlap, deleting one is masked: the others still block the harm, so the gate sees only the gain in task success. Across 411 improvement loops under ten gate rules, a deletion behind another working safeguard "adds almost no observable harm." At equal cost, 20-task batches kept every harness protected while 40-task batches lost all six, because a larger batch lets a significance test approve cheaper deletions. The lost depth became harm when the harness was reused with a weaker model. The GEPA optimizer kept every safety rule verbatim, yet two of its three final harnesses misused credentials under a weaker model. The authors suggest freezing safety text or auditing one-layer ablations.

Instructions hidden in runtime feedback

CARI (arXiv 2610.38799) studies a peer component, such as a backend service, test harness or CI job, that a coding agent depends on but that has no authority over the agent's machine. The attacker embeds instructions in that component's diagnostic output, and the victim agent may act on them while debugging. Across four languages and five models, ten attacks succeeded in 2,845 of 3,000 trials (94.8 percent). Against product agents the rates were 61.3 percent on Codex and 58.7 percent on Claude Code. Generic warnings did not help. Explicit policies stopped attacks but "substantially over-refuse legitimate actions."

Malicious memory files

A study of project memory files such as CLAUDE.md (arXiv 2610.38521) wrote two malicious files for each of 12 CWE classes and ran Claude Code and OpenCode on 36 SecRepoBench tasks. The files changed the rate of secure completions by between −8 and +6 percentage points, "with no consistent pattern." The bigger difference was disclosure: Claude Code named the file in about two thirds of its final messages, while OpenCode mentioned it almost only in its reasoning trace, which the developer does not usually read. The authors call this a pilot and outline the larger studies needed.

Numbers from the wrong run

Re-analysing 540 judgments from a public benchmark of AI research agents and hand-labelling 256 numerical claims, the authors of arXiv 2610.38691 find that failures were mostly misalignment between paper and work, not overt cheating. About four in five unsupported claims matched real values in the agent's files "from the wrong experiment, metric, or run." Their deterministic claim checker reached over 90 percent precision (78 of 85). An LLM verifier found more traceable claims but accepted about three times as many misattributed ones.

Repairs must be scored by the agent that runs them

When a failed agent's step is repaired, which evaluator should pick the repair? arXiv 2610.38610 finds that choosing repairs with a larger model's rollouts lost 0.25 in expected net success on ALFWorld for a Qwen2.5-7B agent, and that "twenty 14B rollouts per candidate are worth no more than two of the 7B agent's own." Repairs should be verified with the executor that will run them.

What remains uncertain. Masked erosion and the repair study use controlled environments. CARI's product-agent numbers depend on the authors' attack set. The memory-file study is small, and its null result on security does not show the files are harmless.

Analysis: The common thread is that a check measures only what it is shown. A gate that sees task success cannot see a lost safeguard. A judge that sees a matching number cannot see that it came from another run. If Jenya's agent tunes the prompts for the Belgavi's Book Store help desk, run each candidate once with each safety rule removed, and fail the change if any single removal causes harm.

Sources and evidence

October 28 preprints: acceptance gates let a self-improving agent delete overlapping safeguards unseen, with the lost depth surfacing as harm under a weaker model (2610.38392); runtime-feedback injection steered Codex and Claude Code in about 60 percent of trials (2610.38799); malicious memory files had no consistent effect on code security but agents differed sharply in disclosing them (2610.38521); most unsupported numbers in agent-written papers came from the wrong run (2610.38691); and repairs should be scored with the executing agent (2610.38610).

Views are my own and do not represent any employer, past or present.