Can you trust coding benchmark scores? TestJack's audit refuted a third of passing trials

When a coding agent “passes” a benchmark task, how often did it actually do what the task asked? A new preprint by Shuangjie Yao, Hao Wang, Koushik Sen, Simin Chen, Baishakhi Ray and Dawn Song (Columbia University, UC Berkeley and George Mason University; arXiv, October 7) tries to measure that gap with a tool called TestJack. It re-checks trials that already passed a benchmark’s tests and asks whether they violate what the task prompt requires. We read the full text, including the appendices; we did not rerun anything, and the page gives no peer-review status. It sits alongside our earlier pieces on sub-agents in coding agents and on security benchmarks, which also lean on benchmark scores.

What is the problem?

Most coding benchmarks call a solution correct if it passes a fixed set of unit tests written when the benchmark was released. The authors argue that such tests check only part of what a prompt requires, so an agent can pass them while missing required behavior, or while gaming the tests. They separate two defects: insufficient tests, where the prompt requires something that no test checks, and an underspecified prompt, where a behavior is only implied and it is unclear whether it must be implemented. TestJack refutes trials for the first kind and, by its own rule, does not refute them for the second.

How does TestJack work?

• Audit. For each passing trial, an audit agent builds three copies of the repository: with the trial’s patch, with the reference solution and with no patch. It breaks the prompt into testable requirements, examines the diff for code the tests never run, for reward hacking such as hard-coded expected values or tampering with the harness, and for other defects, and writes tests for what it suspects.
• Certification. A test counts only if it passes on the reference solution and fails on the trial. A separate reviewer agent then re-judges each test, treating the prompt as the only authority on what is required, and rejects tests that assert behavior the prompt does not ask for. A trial is refuted if at least one valid test fails on it. The authors call such a test a “witness”.
• Who does the work. GPT-5.5 at high reasoning effort, running as a Codex agent inside each task’s environment, with at most three generate-and-validate iterations per trial.
• Lightweight variant. To cut cost, it analyzes each task once, audits a few randomly sampled trials in depth, and replays every confirmed witness on all other trials of that task.

What did they audit?

Benchmark	Tasks audited	Passing trials
DeepSWE	113	2,848
SWE-bench Pro (public split)	731	751
SWE-bench Verified	500	805
SkillsBench (coding tasks, no GPU)	13	55
SWE-Marathon (no GPU, not ultra-long)	2	28

The trials come from six frontier model backends (Claude Opus 4.6, 4.7 and 4.8, GPT-5.5 and 5.6, and Gemini 3.1 Pro). For DeepSWE, SkillsBench and SWE-Marathon the authors used trajectories released by the benchmark creators, from vendor agents such as Claude Code, Codex CLI and Gemini CLI and from open scaffolds. For SWE-bench Verified and Pro they ran GPT-5.6 Sol with medium reasoning effort in mini-swe-agent, two trials per task. That is 8,862 trials in all, of which 4,487 passed the benchmarks’ own verifiers.

What did it find?

TestJack refuted 1,545 of the 4,487 passing trials, or 34.4%, which lowers the resolution rate over all 8,862 trials from 50.6% to 33.2%.

Benchmark	Passing trials refuted	Resolution rate, before to after
SWE-bench Verified	7.6%	80.5% to 74.4%
SkillsBench	18.2%	64.7% to 52.9%
SWE-bench Pro	29.2%	51.4% to 36.4%
DeepSWE	43.5%	45.7% to 25.8%
SWE-Marathon	53.6%	35.4% to 16.5%

• Careful verifiers did not remove the problem. The two highest rates came from DeepSWE and SWE-Marathon, whose verifiers were written from scratch specifically to avoid weak tests.
• Mind the samples. SkillsBench and SWE-Marathon rest on 13 and 2 tasks, and the SWE-bench Verified and Pro numbers come from one model and harness the authors ran themselves.
• An example. On a SWE-bench Pro task for the Navidrome music server, the prompt asks for album image paths to be stored in a new database column and for a full rescan after migration, among six required behaviors. The official evaluation runs one scanner test with a single changed assertion, which checks only the first behavior. A GPT-5.6 Sol trial added the model field but omitted the migration that creates the column, so its code refers to a column the database does not have. It still counted as passing.

How does the cheaper variant compare?

On the three benchmarks where it could be run, with three audit rounds per task, the lightweight variant refuted 1,863 trials against 1,265 for the full audit. It recovered 72.4% of the full audit’s refutations, with a precision of 49.2%, at a cost of about $4,037, 84.6% below the full audit. It did worst on SWE-Marathon (recall 40.0%), where few passing trials share an audit. A static test-strengthening baseline, SWE-ABS, refuted 248 trials, recovered 9.9% and refuted none on SWE-Marathon. The authors say the lightweight approach amplifies noise and ambiguity by spreading one trial’s finding across others, which is why its precision is low. Recall and precision here measure agreement with the full audit, not human labels.

Where is the audit debatable?

The authors are open about it. Their rule treats a requirement that the prompt never states, and that most models skip, as a flaw in the prompt, not in the trials. By that rule, 29 of the 113 DeepSWE tasks (25.7%), 3 of the 13 SkillsBench tasks and both SWE-Marathon tasks have underspecified prompts. One of their examples is a Textual widget flag whose meaning the prompt never defines: all 30 audited trials chose one reading, the reference chose the other, and the witness test failed all 30, though the authors say it does not expose a defect in the trials. They add that skipping is not the same as ambiguity, because test-based rewards give no credit for untested behavior.

What are the limits?

The authors list them: the method needs a reference solution for each task and assumes tasks are well formed; the audit and reviewer agents are LLMs, so a refutation is only as reliable as their reading of the prompt, and checking against the reference guarantees only that the behavior exists, not that the prompt requires it; the lightweight variant trades coverage and precision for cost; and the results cover a fixed set of frontier models and exclude non-coding, GPU and ultra-long tasks. The authors provide an artifact on an anonymized link, which we did not inspect.

What to keep in mind

These are our observations:

• “34% of passes are wrong” is the tool’s refutation rate. Each refutation is backed by a runnable test, but the decision that the test reflects a real requirement is made by LLM agents, and no human-labelled check is reported.
• The size of the effect depends on the benchmark and the setup. The rates run from 7.6% to 53.6%, so a single figure for “coding benchmarks” is misleading.
• Treat a pass as evidence, not proof. If you use agent output in your own pipeline, an acceptance check against the written requirements is a reasonable extra step. This echoes the way a single security score can shift with how a test is written.

Editorial review: October 9, 2026 (AI). Prepared from the full text of arXiv 2610.10619 (v1, October 7; HTML version, including methodology, Tables 1 to 4, appendices A to C and the limitations section). We did not rerun any experiment or inspect the authors’ artifact. All figures are the authors’; the chart is ours, built from Table 2. “What to keep in mind” is ours.