FDE-Bench: can an AI agent deploy software that actually works?

A Dockerfile can build successfully while its application crashes on startup. A Kubernetes manifest can look plausible while its service never becomes Ready. FDE-Bench, submitted to arXiv on September 23, 2026, asks a narrower and more useful question than “can an agent write configuration?”: can its submitted files produce a running deployment that meets a stated contract?

The authors assembled 136 tasks from 149 hand-authored blueprints: 37 single-image, 43 Compose and 56 Kubernetes tasks. There are 66 tasks that start from source and requirements (greenfield) and 70 that start with a deliberately broken configuration (repair). Applications were written for the benchmark, with scenarios informed by real incident reports. This is a test of deployment configuration, not modification of application code or operation of a live production cluster.

Why the replay matters

The agent receives a ticket, a machine-readable specification and a workspace. It can inspect files, edit deployment artifacts, build and test them. At submission, the grader collects only declared artifacts, tears down the running environment, places those files over clean fixtures, and deploys again. A hand-started container or a secret injected only into the live sandbox disappears. This makes the result more meaningful than judging the final terminal state.

The fresh deployment passes four programmatic layers: build, stable readiness, behavior, and specification conformance. Behavior includes endpoint requests, cross-service operations and monitoring checks; conformance includes requirements such as replica counts, non-root execution and no plaintext secrets. Build and readiness gate the last two layers. Each layer contributes 0.25 to the Deployment Score, but a task is resolved only when all four pass. No language-model judge grades the output.

The construction gate adds useful context. For a task to enter the release, its reference solution must resolve it, while an unchanged submission, mechanical transcription of the specification and a generic server stub must fail. Those floors resolved zero tasks in the evaluation. The authors also disclose seven check-to-specification gaps that remain after a revision; they report that none changes a resolution or score in the published grid. Such disclosure helps readers assess the benchmark, but does not certify every checker against every possible shortcut.

What the seven-model comparison shows

Each model used the same four tools and fixed prompt, with one episode per task and task-specific call and time budgets. The table reports the paper’s v1.1 136-task grid. Resolution is all-or-nothing; Deployment Score (DS) gives partial credit for reached layers.

Model	Resolved	Mean DS	Median tool calls
Gemini-3.6-Flash	75.0%	0.857	35
Claude-Sonnet-5	72.8%	0.855	29
GPT-5.6-Terra	69.9%	0.842	9
DeepSeek-V4-Pro	69.9%	0.831	35
DeepSeek-V4-Flash	66.2%	0.811	30
GPT-5.6-Luna	63.2%	0.785	11
Gemini-3.5-Flash-Lite	52.9%	0.719	37.5

Gemini-3.6-Flash has the highest observed resolution rate, 102 of 136 tasks. The paper’s family-clustered comparison does not support a reliable lead over Claude-Sonnet-5 (p=0.59). Its reported 95% Wilson interval is 67.1–81.5%, and there was only one run per model–task pair. The ranking is therefore a result under this scaffold and budget, rather than a general deployment-agent leaderboard. GPT-5.6-Terra’s nine median calls, compared with 29 for Claude-Sonnet-5 at similar resolution, are worth investigating; these counts alone do not establish total cost or time.

The failure that matters most

Across 313 unresolved model episodes, 52 fail at build, 110 at readiness, 97 at behavior and 54 at conformance. This challenges a simple “the manifest compiled, so deployment succeeded” reading. A service must settle into a healthy state before the later checks can even be evaluated. Another 103 of these failures ended at a tool-call or time cap; a cap is an episode termination reason, not a separate failure layer.

Mean resolution is 82.0% for repair and 51.3% for greenfield, a 30.7 percentage-point gap. Every model shows the same direction. Yet the two groups contain different tasks, and repair starts from a mostly complete, intentionally damaged configuration. The result supports the practical value of a useful starting artifact; it does not prove that fixing any particular deployment is inherently easier than creating that same deployment from scratch.

The paper also describes one practicing junior engineer directing Claude-Sonnet-5 through 25 stratified tasks. The engineer resolved 23/25 (92%), versus 18/25 (72%) for the autonomous baseline on those tasks. That is an interesting example of human direction helping on this subset. It combines one participant with a richer Claude Code interface; session time was not logged, and the paired sign test reports p=0.125. It cannot isolate the effect of human judgment or estimate a general human advantage.

How to use the finding

For a deployment agent in a real workflow, our first acceptance test would mirror the paper’s replay idea: rebuild only committed configuration in a clean environment, then check readiness over time, service-to-service behavior, monitoring and explicit constraints. Track which gate fails, not only whether a YAML file parses. Compare new work and repair work separately, and include human review for high-impact deployment decisions.

FDE-Bench is a useful research instrument, not a production safety certificate. Its tickets are in English, its tasks run in a Linux/x86 Docker and single-node Kubernetes setting, and its applications imitate common web stacks. The published intervals vary over tasks, not repeated runs; timed checks can still depend on host load. The paper and its appendix offer the full protocol. The paper links a code repository, but that linked anonymous host returned an access error during this review, so we did not independently execute the benchmark.