Do sub-agents help coding agents? A 2,124-run study of Codex, Claude Code and Kimi Code

Should a coding agent spawn sub-agents on its own? Claude Code’s dynamic workflows, Codex’s multi-agent mode and Kimi Code’s agent swarm all say yes. A new preprint from Han Li, HanHaoNing Li, Ziqian Jiang and Yiling Lou (Nanjing University, Central University of Finance and Economics and UIUC; arXiv, October 7) tests that with a simple switch: the same agent and model with sub-agents on and off. We read the full text; we did not rerun anything, and the page gives no peer-review status. It extends the question behind our look at whether a minimal agent can match elaborate harnesses.

What did they test?

The authors compared three agents, each with its native sub-agent feature: Codex with GPT-5.4 (the multi_agent options), Claude Code with Claude Opus 5 (dynamic workflows, switched on with ultracode) and Kimi Code with Kimi K3 (swarm_mode). Pairs were chosen in July 2026, at the highest reasoning effort, and in the sequential mode the sub-agent tools were disabled so none could be created. Model, prompt, environment and time budget were the same in both modes. The paper gives no software version numbers.

They sampled 354 tasks from five benchmarks, 20% to 25% of each: 100 from SWE-bench Verified (short, bounded issue fixes) and four long-horizon sets that ask for whole repositories, from RepoZero (150 tasks), ProgramBench (50), NL2Repo (26) and LoopsBench (28, sustained multi-step development). That makes 2,124 runs and about 11.2 billion tokens, or $20,780.02 at standard API rates.

Did sub-agents raise the success rate?

Not reliably. Share of tasks fully solved, with sub-agents on / off:

Benchmark	Codex	Claude Code	Kimi Code
SWE-bench Verified (100)	71.0 / 71.0	59.0 / 83.0	59.0 / 77.0
RepoZero (150)	40.7 / 40.0	26.7 / 42.7	38.7 / 37.3
NL2Repo (26)	57.7 / 34.6	65.4 / 57.7	46.2 / 57.7
LoopsBench (28)	25.0 / 21.4	21.4 / 7.1	25.0 / 25.0

No agent fully solved a ProgramBench task in either mode, although sub-agents improved the share of tests passed for Codex and Kimi Code.

• The pattern depends on task length. The gain was largest on LoopsBench, the longest tasks. Averaged over the three agents, the test-pass difference rose from −8.3 points on RepoZero to −0.04 on NL2Repo, +1.2 on ProgramBench and +4.3 on LoopsBench, so larger expected outputs went with better results for concurrency.
• Short tasks got worse. On SWE-bench Verified, Claude Code lost 24 points and Kimi Code 18, while Codex was unchanged.
• Statistics. Pooling the long and short tasks, the authors report that Claude Code’s drop (p ≈ 5×10⁻⁷) and Kimi Code’s drop (p ≈ 0.013) are statistically significant, and Codex’s small gain (p ≈ 0.23) is not.
• Small samples. Claude Code’s rise on LoopsBench, 7.1% to 21.4%, is two tasks solved against six, and the authors count four tasks it solved only with sub-agents. The NL2Repo swings rest on 26 tasks.
• Sub-agents sometimes solve what sequential runs cannot. In most agent-and-benchmark pairs, concurrency produced some tasks solved only in that mode, even where the average did not improve.

What did it cost?

	Codex	Claude Code	Kimi Code
Tokens per task, SWE-bench Verified (M), on / off	3.37 / 1.34	2.14 / 1.02	1.01 / 0.98
Tokens per task, LoopsBench (M), on / off	51.65 / 11.45	21.73 / 15.52	41.49 / 11.29

• Tokens: 1.41 to 3.31 times the sequential level across the agents. Counting failed runs against solved tasks, Codex used 24.12 million tokens per solved task with sub-agents against 7.69 million without.
• Time: parallel work did not make runs faster on average. Run time rose in 14 of the 15 agent-and-benchmark pairs; Claude Code took 27.8 minutes per SWE-bench Verified task with sub-agents against 17.5 without.
• When it did save time: of the 366 tasks solved in both modes, concurrency was faster on 102 (27.9%) and at least 20% faster on 46 (12.6%).

How do the agents delegate?

Sub-agent counts grew with difficulty: on average 6.69 created per easy task, 7.53 per medium and 15.84 per hard. On the hardest LoopsBench tasks Claude Code created 133.69 sub-agents per task and Kimi Code 67.00, while Codex kept small groups (mean peaks of 1.21 to 3.00). Codex started delegating earliest, at a median of 4.0% of the run against 22.1% for Claude Code and 26.0% for Kimi Code, but 24.3% of its runs created sub-agents that never actually overlapped. Claude Code sustained overlap longest, with a median of 52.7% of the run.

Why did it fail?

The team annotated all 1,062 concurrent runs by hand, each by two of three annotators (Cohen’s κ = 0.77), and found failures in 650 of them, 804 instances in all. Their taxonomy has four categories, 13 subcategories and 28 patterns:

• Shared state and merge, 33.2%: mostly concurrent writes, such as one agent overwriting another’s deliverable (120 cases).
• Execution governance, 29.0%: results arriving at the wrong time, mostly sub-agents terminated before finishing (128 cases).
• Task orchestration, 28.5%: load imbalance, mainly fan-out that exhausts the time budget (154 cases).
• Global context, 9.3%: sub-agents briefed incompletely, or work declared done without verification.

One example in the paper: Kimi Code split a rebuild of the seqtk tool across 20 sub-agents in a 40-minute budget, assumed a two-hour timeout and waited for the whole batch, so integration never happened and the program passed 92 of 440 tests.

When did it help?

Looking at 131 pairs where concurrency won, the authors found three conditions: complementary work, such as eight sub-agents handling files and checks while the main agent wrote the core algorithm (734 seconds down to 468 in one Kimi Code case); alternative attempts, where two builds were compared and combined (Codex passed 292 of 391 tests against 0 sequentially); and independent validation, where a sub-agent’s finding corrected the main agent. They advise enabling concurrency selectively, for hard, long, decomposable tasks, and weighing the cost.

What are the limits?

The authors name them: the results are scoped to three agents and five benchmarks; runs are stochastic and each condition was run once because of cost, though their sequential Codex numbers on LoopsBench are close to those first reported for that benchmark; and manual annotation can be subjective. They released the trajectories, codebook and scripts on GitHub, which we did not inspect.

What to keep in mind

These are our observations:

• It measures defaults at one moment. The agent and model pairs date from July 2026 and the tools change quickly, so treat these as a snapshot, not a ranking of products.
• It is not a verdict on any one agent. Claude Code’s large drop is on a 100-task bounded sample at one run per condition, and the same agent gained on the longest tasks.
• For everyday use, the cost side is the clearest signal. Even where success held, token use rose several-fold. The finding that concurrency pays on long, decomposable work and not on small fixes fits what the EPFL and Apple study saw with parallel agents, though the tasks and models differ. Sub-agent features also widen the surface that security benchmarks for coding agents examine.

Editorial review: October 8, 2026 (AI). Prepared from the full text of arXiv 2610.10263 (v1, October 7, read through the HTML version including the tables, taxonomy, benefit analysis and threats to validity). We did not rerun any experiment, inspect the released artifact or test the agents ourselves. All figures are the authors’; the chart is ours, built from Table 3. “What to keep in mind” is ours.