Can AI review code without humans? CRBugBench finds the best agent catches 71.9% of known bugs

If an AI agent writes the code and another reviews it, how many bugs does the reviewer actually catch? A new preprint puts a number on it. Chenyang Yang, Hao He, Ti-Chung Cheng, Jenna Butler, Emerson Murphy-Hill, Carmen Badea and Christian Bird (Carnegie Mellon University, Microsoft and Microsoft Research; arXiv, October 9) built CRBugBench and found that the best review agent catches 71.9% of known bugs in its top five findings, with a sharp drop on proprietary code. We read the full text and its tables; we did not rerun anything, and the page gives no peer-review status. It sits next to our earlier pieces on sub-agents in coding agents and on auditing coding benchmarks.

What is CRBugBench?

A benchmark that asks whether a review agent finds bugs a human has already confirmed. The authors start from the idea of a “lights-off” software factory, where AI writes and reviews code with little or no human supervision, and argue that most code review benchmarks compare an AI’s comments with human comments, which are an unreliable ground truth. CRBugBench instead takes bugs that humans verified and traces them back to the pull request (PR) that introduced them. The review agent sees only that PR in a sandboxed repository, is not told a bug exists, and writes a ranked list of findings.

How was it built?

From repository history, in two streams, and then audited. The authors picked established repositories in Python, Rust, C#, JavaScript/TypeScript and Java (more than 2,000 PRs and 200 contributors each, one repository per application domain per language). The pre-merge stream starts from 304,601 merged PRs with 1,436,120 review threads, filters to 472,039 candidate threads, samples 90 per repository and keeps 865 vetted bugs after an LLM filter. The post-merge stream starts from 195,075 bug-fix PRs, ends with 689 fix and introducing-PR pairs traced with the SZZ algorithm and filtered by LLMs, and each stream is sampled down to 500 tasks. That gives 1,000 open-source tasks across 98 repositories, plus 800 tasks from two industry monorepos at one large company (averaging over 100,000 source files against about 3,000 in the open-source repositories). A hand check of 40 sampled bugs found 37 valid (92.5%, 95% interval 80.1 to 97.4%).

Scoring uses a panel of three LLM judges (gpt-5.5, claude-opus-4.8 and gemini-3.1-flash) and a majority vote; against 40 samples annotated by humans, agreement was kappa 0.89. The metric is recall@k, the share of tasks where the known bug appears among the agent’s top k findings. The main harness was copilot-cli 1.0.74 with its official /review prompt and the highest reasoning effort available. Agents ran in a Docker snapshot with internet access restricted and an instruction against cheating, and an AI-assisted audit of trajectories found no cheating, the authors say.

How many bugs did the agents catch?

The best agent found about 72%, up sharply in five months. Within its top five findings, claude-opus-5 caught 79.6% of the bugs humans found before merge and 64.2% of those found after merge, which averages to 71.9%. The authors report 47.7% for claude-opus-4.6 and a rise to 71.9% from February to July 2026. It cost $2.97 per task on average, with a mean latency of 9.8 minutes, and 89% of the bugs that were ever found appeared in the top three items.

Cheaper models gave good value. gpt-5.6-luna matched about 78% of claude-opus-5’s performance at about 21% of the cost. The authors also compare agents with humans, with a caveat built into the benchmark: because the tasks were built from a mix of bugs humans caught before merge and bugs that escaped to later fixes, humans alone catch about 50% by construction, the best agent 71.9%, and the two combined 82.1%. The 50% is not a general rate for human reviewers.

Why do the agents miss bugs?

Mostly because the bug is not visible in the diff, and because agents read more than they run. In a regression across all reviews, larger diffs (odds ratio 0.591), more heterogeneous diffs (0.795) and bug-fixing PRs (0.521) lowered the chance of a catch, as did usability bugs (0.454), while bugs visible in the diff were far easier (2.621) and severe bugs somewhat easier (1.906). Language and repository size had no significant effect once diff size was accounted for. Humans were better at maintainability issues (2.702) and worse than agents on more severe bugs (0.450) and reliability issues (0.591).

On behavior, agents typically began with git operations, then spent about 75% of their turns reading (39%) and searching (36%), and about 8% executing code. claude-opus-5 executed code in 9.7% of turns against 6.9% for gpt-5.6-sol, and it stayed in the loop longer: half of its runs ended by turn 35, against turn 17 for gpt-5.6-sol. Of all runs, 95.6% inspected the file holding the bug, 80.2% produced a finding about it and 56.2% hit the bug; for claude-opus-5 those figures were 91.9% for a finding and 75% for a correct diagnosis. The authors describe cases of overtrust: in a Selenium task every model endorsed the buggy condition because both a code comment and the PR description asserted the intended behavior, and in an Elasticsearch task claude-opus-5 cited the defective line as its standard of correctness.

What happened on proprietary code?

Performance fell by up to 25.5%. The proprietary tasks contain far fewer bugs visible locally in the diff (22.8% against 46.1% on GitHub) and many more that need project context, domain knowledge or runtime evidence. The authors say adjusting for bug visibility alone explains about 20% of the gap. They add that proprietary code is certainly not part of any model’s training data, which makes it a useful cross-check on the public results.

What bought more recall, and what did it cost?

Prompting was the cheapest lever, and effort and agent count were expensive. The authors report that asking agents for more comprehensive findings and then ranking by severity gained up to 5.8% at 1.10x the cost; for gpt-5.6-sol the average gain was 5.3%, but for claude-opus-5 it was small or absent. Raising gpt-5.6-luna’s reasoning effort from low to maximum moved it from about 36% to 57% at 23.22x the cost. Multi-agent setups gained 4.7% at 5.55x in the introduction and 2.8% on average at 3.68x in Section 4.6; the paper does not say, but we read these as the strongest setup and the average. In the multi-agent table for gpt-5.6-sol at recall@5, sub-agents lowered recall on the pre-merge track by 3.7 points and raised it by 6.0 on the post-merge track.

Harness choice also mattered: copilot-cli outperformed claude-code in the authors’ tests. Pairing claude-opus-5 with claude-code sharply raised cost, because it used about 9.0 sub-agents per run against none in copilot-cli, which the paper links to Anthropic’s documentation saying claude-opus-5 delegates to sub-agents more readily than earlier models. The authors tell developers to watch for such cost surges when a model changes.

So can a software factory run lights-off?

Not yet, in the authors’ view. They write that the present evidence supports lights-off operation only for narrowly scoped, low-consequence PRs, and that for consequential changes agents are better treated as scalable peers with an accountable human making the final call. Their recommendations: scale review effort to PR complexity, steer agents by project priorities (for example asking explicitly about usability or maintainability), supply repository knowledge, dispatch models by PR characteristics, and monitor review quality as models change. They also suggest teams build their own benchmark from their repositories using the paper’s method.

What are the limits?

The authors list them. The benchmark measures recovery of known bugs, not overall review usefulness. Ground truth can be wrong (the audit found 3 of 40 sampled tasks invalid), and the judges are LLMs, including models from the families being tested, mitigated by a three-model panel. Agent runs are stochastic, the regressions are observational and show association, not cause, and the selection favors mature, active, popular projects, so results may differ for smaller or younger projects, uncommon languages and configuration-heavy code. The authors are at Microsoft, Microsoft Research and Carnegie Mellon, the main harness is Microsoft-linked copilot-cli, and the proprietary repositories belong to one unnamed large company; we mention this as context, not as a finding.

What to keep in mind

These are our observations:

• Recall of known bugs is a floor, not a verdict. An agent can find bugs nobody had recorded, and can also add noise; the benchmark does not score either.
• Read the 71.9% with its mix. It averages a pre-merge and a post-merge track and counts only the top five findings. The post-merge track alone is 64.2% for the best agent.
• Your repository is the benchmark that matters. The proprietary drop and the authors’ advice to build custom benchmarks both point to measuring on your own history before trusting a score.
• Watch cost as well as quality. Effort and multi-agent settings multiplied cost several times for modest gains, and one model-harness pairing changed spending through sub-agent use.

Editorial review: October 12, 2026 (AI). Prepared from the full text of arXiv 2610.13668 (v1, October 9; HTML version, including Tables 1 to 3 and the discussion). We did not rerun any experiment, did not look for a replication package and did not check the authors’ claims independently. Per-model averages and cost per task in the chart are our computations from Table 1; all other figures are the authors’. “What to keep in mind” is ours.