SWE-Bench Performance Across Agentic Coding Frameworks

SWE-bench performance across agentic coding frameworks measures more than the model. It measures the model, the machinery around it, and the conditions of the test. The scores become useful only when those parts are separated. Otherwise, a leaderboard can turn an engineering comparison into a contest between numbers that happen to share a percent sign.

What SWE-bench actually measures and how it works

Princeton researchers created SWE-bench from real GitHub issues and pull requests across popular open-source Python repositories, including Django, Flask, and scikit-learn. A task gives the model an issue description and the repository as it stood when the issue was filed. The required output is a patch. That patch must make previously failing tests pass without breaking tests that already passed, with no access to the evaluation test cases themselves.

Each task runs in an isolated Docker container, and the headline metric is % Resolved, reported as Pass@1. It counts successfully resolved tasks. The distinction is practical. Unlike isolated function-completion puzzles, the fail-to-pass format joins several parts of engineering work, including understanding a bug report, navigating a codebase, and producing a working change. That broader workflow helped make SWE-bench the de facto coding benchmark before 2026.

The three variants and what separates them

SWE-bench Verified, SWE-bench Pro, and Senior SWE-Bench test different task populations under different conditions. They are not interchangeable difficulty settings. Verified tasks are well-specified, mostly single-file Python bug fixes. Pro tasks span multiple languages, involve many more changed lines on average, and demand work over longer horizons. A score on one therefore cannot be read as the equivalent score on another.

Pro also has two distinct evaluation contexts, namely the SEAL public set with standardized scaffolding and the private commercial set. Vendors report their own results separately, with much higher numbers. The distinction is easy to lose in a headline. A public-set result with shared tooling answers a different question from a vendor’s result with its own setup, while private code changes whether the model could have seen it during training.

Senior SWE-Bench changes the judging method as well as the task population. It uses a taste judge calibrated against human reviewers and a validation agent that writes behavioral tests adapted to each submitted solution. Test passage is not the whole verdict. The benchmark also judges solution quality, making it a test of expert-relevant work rather than simply another collection of bugs with harder tests.

Why the harness around the model is often the primary variable in any score

A harness determines what a model can do between receiving an issue and submitting a patch. It can let the model explore file trees, run tests repeatedly, retry failed patches, and follow test-driven loops. Those abilities belong to the system that gives the model tools, time, and opportunities to correct itself.

That changes what a successful attempt can contain. Pass@1 does not mean that the model necessarily wrote its final patch in one uninterrupted response. A scaffold may support several rounds of investigation and revision within the evaluated attempt. Retry budgets and tool calls can thus change the result even when the base model stays the same. The harness is part of the contestant.

One reported Verified submission using aggressive scaffolding scored a double-digit margin above the same base model’s standalone result of 69%. That gap led to debate over both scaffolding and contamination. Without isolating how much each contributed, the gap shows why assigning the entire result to the model is unsound, because the surrounding system changed substantially.

By giving models identical tooling and a 250-turn limit on Pro, Scale’s SEAL harness addresses that attribution problem. Standardization removes an important source of variation, though it does not establish which harness would suit every model best. The SWE-Serve paper supplies a useful counterweight to the assumption that specialized tooling must improve results. It found that switching to model-specific harnesses did not improve mean Pass@1 in either evaluated configuration.

What the Verified-to-Pro gap reveals when the same model is tested under both

The sharp fall from Verified to Pro for the same model is not evidence that the model regressed. The test changed beneath it. Pro has harder tasks, and the conditions of the test can differ from a vendor’s Verified setup. To tell whether task difficulty or scaffolding caused the whole drop, the other conditions must stay the same.

This is where the distinction between three reporting contexts becomes essential. Verified vendor-reported, Pro vendor-reported, and Pro SEAL-standardized results describe different combinations of model, task population, and tooling. A comparison across them can reveal how sensitive performance is to the setup. It cannot, by itself, measure a pure change in model capability.

The available comparison establishes a substantial gap without supplying a complete matched table of exact model scores. Precision should stop there. The supported conclusion is structural, because strong Verified results can reflect scaffold engineering and contamination as well as coding ability. For model comparisons, the practical rule is to use results from one scaffold measured at one time. A launch-post score beside a standardized-board score compares two pipelines.

The current leaderboard picture across Verified and Pro, read correctly

The leaderboard picture is best read as three boards rather than one ranking. Verified vendor reports show how models perform under the vendor’s own test conditions. Pro vendor reports use the harder set of tasks with the vendor’s own scaffold. Pro SEAL results use the same tooling for all models. Only the last provides that controlled basis for comparing models, and it still describes performance under a particular harness.

Benchmark version is another necessary boundary. Scale AI announced SWE-Bench Pro V2 on September 22, 2026, describing a refreshed public split, a modified benchmark, and a locked evaluation protocol co-developed with Reflection. A result from before that release cannot silently become a V2 result. The September 28, 2026 leaderboard snapshot identifies the SEAL public evaluation as Pass@1 over 731 tasks, but does not provide a full set of model-by-model scores here. With those scores missing, it cannot show exactly how Verified, Pro, and open-weights entries rank now.

Verified’s leading models occupy a narrow band, weakening its value for separating frontier systems. Open-weights results from a standardized scaffold should be compared with other models using that same scaffold, with unrelated vendor claims kept separate. Senior SWE-Bench reports Claude Opus 4.8 in the lead, but at a low score. Leadership on that harder test should not be mistaken for broad mastery of expert-level engineering.

Named open-source harnesses and what each one is actually built for

SWE-agent, from Princeton, is the canonical research reference implementation for agent-computer interfaces. Its design is minimal and text-first. It also forms the lineage from which most later harnesses descended, making it a reference point for understanding how models interact with coding environments.

OpenHands, from AllHands, is the production-oriented open scaffold. It combines Docker sandboxing with active weekly releases and a large community. The reported results give it the best Verified score of any open scaffold covered in the source material. That distinction is limited to Verified and says nothing about whether it leads on every benchmark, workload, or operating constraint.

The split explains why an open-source harness comparison needs more than a score column. Research reference designs and production-oriented scaffolds are built for different uses, even when both can run benchmark tasks. Their structure is part of the comparison.

Commercial coding agents: what their reported scores include and exclude

Commercial coding agents present a closed pipeline in which the vendor controls the model weights, scaffold, tool permissions, and compute environment. Their reported scores therefore describe a bundled system, offered with its own infrastructure and pricing. Those parts cannot be kept the same across vendors. Directly comparing such a result with an open-scaffold entry obscures what produced any advantage.

For purchase decisions, that makes the private-codebase subset of Pro a more relevant signal than a Verified headline alone. It tests code the model could not have encountered during training, matching the unfamiliarity of a team’s proprietary repository. A system that succeeds on public code has not necessarily demonstrated the same ability on inaccessible code.

SWE-bench does not establish security vulnerability detection, code review quality, maintainability, compliance with organizational coding standards, or performance under production load. How well a model works in non-Python languages is also a gap in Verified’s scope, so these gaps mean that a high resolution rate cannot answer those purchasing questions by proxy.

Where SWE-bench scores stop predicting real-world performance

Detailed issue descriptions reduce the ambiguity of diagnosis, while mostly single-file Python fixes narrow the space of possible changes. Real work is less accommodating. It can be ambiguous, multi-file, multi-language, and novel, with requirements that extend beyond repairing an already described failure.

FeatureBench provides a direct test of that mismatch. The FeatureBench paper measured Claude Opus 4.5 at a 74.4% resolved rate on SWE-bench, yet found that it succeeded on only a small fraction of FeatureBench tasks. Those tasks call for building complex features and come from 24 open-source repositories. The contrast is substantial. A strong bug-resolution result did not transfer to the work of developing complex features.

SWE-EVO exposes a related weakness in sustained reasoning. Its evaluation found that GPT-5 with OpenHands achieved 21%, compared with a substantially higher result on Verified. The benchmark tests work on codebases that change over time and requires models to keep reasoning across multiple files. Senior SWE-Bench adds an expert-calibrated quality test to this picture. Its low leading score reinforces the distinction between passing existing tests and delivering a solution that meets senior engineering expectations.

These findings also explain how a lower-ranked Verified system could outperform a leaderboard winner on work humans value. The task distributions need not reward the same strengths. Near-saturation makes Verified especially weak as a fine-grained frontier ranking, although it remains useful for filtering out clearly weaker models. SWE-bench still offers the most realistic coding benchmark of its kind. Its value comes from testing a substantial engineering workflow.

How to read a SWE-bench score without being misled by it

The practical reading starts with four questions. Which variant was used, which scaffold ran it, what is known about contamination, and what result is being used for comparison. Those details are rarely stated together. Without them, a precise percentage can conceal an imprecise claim.

The three-board check supplies the next distinction. Verified vendor-reported scores come with contamination risk, and OpenAI has deprecated Verified. Pro vendor-reported results use the vendor’s own scaffold, while Pro SEAL-standardized results use the same tooling to compare models. Contamination status still needs to be tied to the actual task set. The private commercial subset offers the strongest separation from training exposure because its proprietary code is legally inaccessible.

For model selection, Verified remains a rough filter for frontier systems. Within the top tier, small margins fall below measurement noise once scaffold differences and contamination are accounted for. When businesses test models, they should give more weight to Pro’s private commercial subset. Long-horizon and feature-development decisions should also draw on FeatureBench, SWE-EVO, and Senior SWE-Bench, because a bug-fix score does not predict those abilities reliably.

The same interpretability problem follows an agent into production. An outcome combines what the model did with what the scaffold permitted, and a final score does not separate the two. Understanding an agent requires seeing its behavior in relation to its tools and constraints.