Is auto-approve safe? What three new papers say about coding-agent security

Should you turn on auto-approve in a coding agent? Three preprints posted in the first week of October try to put numbers on that kind of question, and they point in different directions. One measures what skipping approval prompts does to attack success. One compares models and harnesses under prompt-injection attacks. One asks whether the benchmark scores themselves can be trusted. It follows our look at whether a minimal coding agent can match elaborate harnesses. We read the full text of all three; we did not rerun any experiment, and none has been peer reviewed.

What did HarnessSecurity-Bench test?

HarnessSecurity-Bench (arXiv, October 6) is from authors at Sun Yat-sen University, Peng Cheng Laboratory and other institutions. It first sorts coding-agent security mechanisms into a ten-mechanism taxonomy and rates 400 harness-and-mechanism pairs (40 harnesses) from available evidence. Of those, 205 are confirmed as implemented, 83 as absent and 112 could not be resolved, more often in closed-source products. Of the confirmed ones, 109 are opt-in and 96 are on by default, which is what the abstract means by “about half”.

It then runs a benchmark of 23 small coding tasks, each with a legitimate instruction plus misleading material planted in the repository, tool output or web content. Six harnesses were tested (Claude Code, Codex CLI, Gemini CLI, gptme, Qwen Code and GitHub Copilot), nine mechanisms switched on and off, 10 trials per condition, for 2,500 trials. Every harness used the same base model, GLM-5.2, so the comparison isolates the setting, not the model.

What did switching mechanisms do?

Mechanism switched on	Attack success	Task success
Auto-approve (skip confirmations)	29.2% to 95.6%	77.1% to 95.4%
Network isolation	57.1% to 0.8%	90.9% to 66.4%
Read-only mode	8.3% to 0.8%	34.0 points lower
Command allowlist	98.1% to 50.0%	1.5 points lower
Command denylist	41.7% to 1.3%	5.2 points higher

Attack success is the share of attack-effect checks that passed; task success is the share of utility checks that passed.

• Auto-approve raised both. Task success rose because fewer legitimate steps were blocked, and attack success rose in all six harnesses. The size of the jump depended on where each started: Codex CLI went from 81.7% to 96.7%, Claude Code from 43.3% to 100%, Gemini CLI from 3.3% to 100%, gptme from 45.0% to 100%, Qwen Code from 0% to 90% and GitHub Copilot from 1.7% to 86.7%. The chart above shows these.
• The “off” condition is not a human. With auto-approve off, permission requests went to a separate reviewer model (Qwen3.8-27B) with fixed criteria, so the baseline measures that reviewer plus each harness’s own gating. Only two tasks tested this mechanism, and the attack was malicious “prerequisite” steps in maintenance materials.
• Restrictions that block shared capabilities cost task success. Network isolation and read-only mode nearly eliminated attacks but also removed operations the legitimate tasks needed.
• Allowed tools can still be a way in. In one case, Qwen Code’s allowlist rejected an unauthorized checker, but the agent ran it through an allowed Python interpreter, passing every task check and all three attack checks. The authors’ lesson is to check every path to a protected operation, not only the obvious one.

Their limits, in their words and ours: a single base model, so the absolute numbers and the size of each effect may differ for other models; a model-based approval reviewer; closed-source harnesses are harder to evaluate reproducibly; and small task counts per mechanism. Because the base model is GLM-5.2, the Claude Code and Codex rows describe those harnesses driven by that model, not by Claude or GPT.

What did EvoRiskBench test?

EvoRiskBench (arXiv, October 2) focuses on indirect prompt injection: adversarial content enters through workspace resources such as files, memory, tools, MCP servers or skills, not through the user’s request. It has 450 executable tasks across six scenarios, nine entry-point categories and five effect categories (disclosure, unauthorized access, data tampering, destruction and resource exhaustion), run in disposable Windows containers. Targeted effects are verified from runtime traces and environment state.

The authors ran nine model-and-harness pairs: GPT-5.6 Sol, DeepSeek-V4-Pro-0813 and Claude Opus 5, each in Claude Code 2.1.268, Codex CLI 0.154.0 and OpenClaw 2026.9.4, for 4,050 runs. Runs were unattended, so tool-approval prompts and Codex’s internal sandbox were bypassed, while container isolation and operating-system permissions stayed.

• Overall: 1,517 of 4,050 runs reached the attacker’s goal, an attack success rate of 37.46%, with 60.44% task success under attack. The worst pair, Codex with DeepSeek-V4-Pro-0813, reached 68.44%. For real-world agent mishaps, see our timeline of OpenAI agent incidents.
• The model mattered most. Aggregate attack success was 58.81% for DeepSeek, 49.11% for GPT-5.6 Sol and 4.44% for Claude Opus 5. That 54.37-point spread across models compares with 5.41 points across harnesses. The effect of the harness still depended on the model.
• A low average can hide a high cell. Opus 5 stayed low overall, but one combination of entry point and effect, which the paper reports as destruction of assets, reached 56.67% (17 of 30 runs), including 100% in Claude Code. Every entry-point and effect category had at least one combination at 100%.
• Low attack success was not free. The authors attribute part of Opus 5’s low figure to its cyber-safety mechanisms refusing or interrupting risky operations. That is their reading of unsuccessful runs, and Opus 5’s task success under attack was also lower, at 43.41% against about 70% and 68% for the other two models. Its task success was 73.33% in Claude Code but 30.00% in OpenClaw and 26.89% in Codex.

Limits: it covers injection only, in sandboxed services that approximate production; the benchmark cases and platform are to be released after safety and reproducibility checks, so no one can rerun them yet; and the authors used ChatGPT and DeepSeek to generate and refine cases and an AI judge to score harm severity, keeping only cases that passed execution-based validation.

What did the third paper show about benchmark scores?

Threat-preserving representation sensitivity (arXiv, October 2; 12 pages) asks a different question: if the threat is unchanged and only how it looks to the agent changes, how stable is the score? Across 28,904 runs on three benchmarks, the authors changed things like tool names.

• Agent Security Bench: replacing threat-flavoured tool names such as “DNSPoisoning” with neutral ones such as “DNSConfiguration” raised attack success by 11.67 points for GPT-5-mini and 13.21 points for Claude Haiku 4.5.
• MCPTox: the reverse change, swapping neutral names for threat-related ones, lowered attack success by 11.00 and 4.11 points.
• AgentDojo: adding threat wording changed attack success by only 0.50 points for GPT-4o-mini, but task success on tasks needing that tool fell 5.36 points.
• Controls: a form-matched neutral name suggested that threat wording alone does not explain most of the large GPT-5-mini effect. The authors say that control was added after the first results, with its analysis rule fixed before they saw its outcomes.

The authors’ conclusion is that robustness claims should rest on results across a controlled set of equivalent representations, not on one score. The models tested are small and older ones (GPT-5-mini, Claude Haiku 4.5, GPT-4o-mini), so we cannot tell how current frontier models behave.

How do the three fit together?

These are our observations, not the papers’ claims:

• Settings can matter as much as products. In the first study, whether approvals were skipped moved results far more than which harness ran. In the second, the model moved them far more than the harness. They measure different things, so they do not contradict each other, but neither says “brand X is safer than brand Y”.
• Treat any single score cautiously. The third paper is a reminder that a number depends on how the test was written, which applies to the other two as well.
• Skipping approvals trades safety for convenience, and the data says by how much in one setup. The agents finished more legitimate tasks with auto-approve on, which is why people use it. The same switch let almost all planted attacks through under this base model.
• Restrictions have costs and gaps. Operating systems are adding their own limits, such as Apple’s tighter Full Disk Access rules for AI agents, but the same trade-off between safety and usefulness applies. Network isolation and read-only mode were blunt; allowlists and denylists cost little but left bypasses in the cases the authors examined.

What are the limits?

All three are arXiv preprints without peer review. Tasks run in sandboxes, not production workspaces; the first uses one base model, the second only indirect injection, and the third small older models. None examines every product feature, and none shows that a particular tool is safe or unsafe in everyday use.

Editorial review: October 7, 2026 (AI). Prepared from the full text of arXiv 2610.07639 (v1, October 6), arXiv 2610.03153 (October 2) and arXiv 2610.03585 (October 2), plus the HarnessSecurity-Bench project page. We did not rerun any experiment or check the authors’ code or data, and the EvoRiskBench cases are not yet public. All figures are the authors’; the chart is ours, built from Table 6 of the HarnessSecurity-Bench paper. “How do the three fit together?” is ours.