Measurements · 2026-09-05 · 1,372 words · 6 min read

SWE-bench explained: what a resolve rate measures and what it misses

How a SWE-bench score is produced, what Verified changed, and five papers on harness effects, lucky passes, contamination, and realistic prompts, with a dataset.

swe-bench benchmarks evaluation agents

A SWE-bench percentage is the most quoted number in agentic coding, and it is a narrower measurement than the way it is quoted. This piece explains exactly what a run does to produce the number, what the Verified subset fixed, and what five later papers found when they looked at the same score from other angles. The dataset ships every number used here with its source, and nothing else.

What a run does

The benchmark comes from the 2023 paper by Carlos E. Jimenez, John Yang, and colleagues (arXiv:2310.06770). Its abstract describes "2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories," and reports that "The best-performing model, Claude 2, is able to solve a mere 1.96% of the issues." Each problem is an issue text, a repository at a base commit, and a hidden set of tests.

The SWE-bench evaluation guide states the procedure in one sentence: "SWE-bench evaluates models by applying their generated patches to real-world repositories and running the repository's tests to verify if the issue is resolved," inside "a containerized Docker environment to ensure consistent results across different platforms." A submission is a JSONL file, one object per instance:

{
  "instance_id": "repo_owner__repo_name-issue_number",
  "model_name_or_path": "your-model-name",
  "model_patch": "the patch content as a string"
}

The command is:

python -m swebench.harness.run_evaluation \
    --dataset_name princeton-nlp/SWE-bench_Lite \
    --predictions_path <path_to_predictions> \
    --max_workers 8 \
    --run_id my_evaluation_run

The definition of success is in the guide's results table: "Instances resolved: The patch made the required tests pass." Which tests are required comes from two columns in the dataset. The Verified dataset card defines FAIL_TO_PASS as "A json list of strings that represent the set of tests resolved by the PR and tied to the issue resolution," and PASS_TO_PASS as "A json list of strings that represent tests that should pass before and after the PR application." Resolved means the first set flips to passing and the second set stays passing. The card also defines patch as "The gold patch, the patch generated by the PR (minus test-related code), that resolved the issue," and test_patch as "A test-file patch that was contributed by the solution PR."

So the number is: the fraction of instances where the agent's patch, applied at base_commit, makes the maintainers' own tests for that issue pass without breaking the others. It is a pass@1 unless a submission says otherwise; the harness scores one patch per instance.

What Verified changed

The repository README at commit 02e7a74ffd0b (2026-09-02; 5,781 stars, MIT, read 2026-09-05) records the milestones: Docker-based evaluation in June 2024, SWE-bench Verified in August 2024 as "A subset of 500 problems that real software engineers have confirmed are solvable," and, on 2026-09-01, SWE-bench Multimodal v2 with 480 tasks. The Verified dataset card describes its 500 rows as "a subset of 500 samples from the SWE-bench test set, which have been human-validated for quality." The point of validation was that some original issues were underspecified or had tests that a correct fix could still fail; the subset removed those. Most quoted scores since late 2024 are on Verified, and a score on Verified is not comparable with one on the full test set or on Lite.

The leaderboard site lists Verified, Multimodal, Multilingual, Lite, and Full tabs (read 2026-09-05; I do not quote scores from it, because they change and this piece would go stale). What the leaderboard requires is documented in the experiments repository, which "contains records of submissions to the SWE-bench leaderboard." A submission needs all_preds.jsonl, per-instance logs with the patch, evaluation report, and test output, and "Human-readable" reasoning traces that reflect "the intermediate steps your system took that led to the final solution." Verification is by re-derivation: "swebench submit verify re-derives every verdict from test output," with "no Docker, no re-execution." That means the leaderboard checks that your logs support your score; it does not rerun your agent.

What the score does not carry

The harness. Sydney Lewis's paper (arXiv:2608.26218) held the model and tasks fixed and changed only the harness's context management: the control showed the full conversation chronologically, the treatment kept the same records but "progressively condensed older tool outputs as context limits approached." On 169 Verified tasks with a 20,480-token window and 480-second attempts, the fail-to-pass fraction moved from 28% to 49%, and complete solutions from 43 to 72. Same model, same tasks, 21 points. A leaderboard row names a system, and a system is a model plus a harness, so a row cannot be read as a model score. This is also why the SWE-agent paper (arXiv:2405.15793) mattered: its 12.5% pass@1 on the full set was an interface result as much as a model result, as discussed in the SWE-agent reading.

Whether the pass was earned. AgentLens (arXiv:2605.12925) analyzed 2,614 OpenHands trajectories from eight model backends on 60 Verified tasks and found that "Among passing trajectories in this subset, 10.7% exhibit behavior we call a Lucky Pass," meaning regression cycles, blind retries, missing verification, or temporally disordered steps. The per-model range was 0.5% to 23.2%, and "some models move by as many as five rank positions when ranked by quality score instead of pass rate." The score counts the patch, not the path.

Contamination. Prathifkumar, Mathews, and Nagappan (arXiv:2512.10218) tested file localization and found models performed "3 times better" on Verified than on alternative benchmarks and were "6 times" better at naming the edited files without context, which they read as evidence that Verified may have been seen in training. A score on public data has a memorization component that is not separable from the number.

The prompt. RealSWE (arXiv:2608.27831) compared real user requests with benchmark problem statements and found that "88% of real prompts but just 7% of benchmark problems" contain only a bare problem statement, and "87% of real prompts are casually written whereas 94% of benchmark problems are formal." Rewriting 381 task families in realistic styles cut resolution by 6.4 percentage points on average across seven models. A benchmark issue is a better-specified task than the one you will type, a point spec-first prompting turns into practice.

Difficulty ceiling. SWE-Bench Pro (arXiv:2509.16941) was built because Verified saturated; it has "1,865 problems sourced from a diverse set of 41 actively maintained repositories." Its abstract reports no resolve rate, so I report none.

Cost. Nothing in the resolve rate says how many tokens, dollars, or minutes a pass took. The FailFast paper (arXiv:2608.03222) shows how much slack is there: an early-stop monitor saved "14.6%-20.4% of execution tokens at a target 5% false-positive rate" on Verified with no change to the task. Two systems with the same score can differ by that much in cost, and the method in what one task costs is the missing axis.

Two numbers, two years apart

The SWE-agent paper reported 12.5% pass@1 on the full 2,294-instance test set in 2024. The mini-swe-agent README at commit 04d809ceab9d claims "Scores >74% on the SWE-bench verified benchmark" in 2026. Those are different benchmarks, different sizes, different harness philosophies (fifteen tool bundles versus bash only), and the second is a README claim I did not reproduce. The change is real and large, and the comparison is still not a comparison. The dataset records both with their source_type, so the paper number and the README number are never mistaken for the same kind of evidence.

The dataset

_README for swe-bench-explained-what-the-numbers-measure.csv
name         the benchmark, result, or finding
kind         benchmark | benchmark subset | result | result claim | finding
size         the number as published (count, percent, or multiplier)
unit         what the number counts; percent rows say "percent resolved" or "percent fail-to-pass"
languages    programming languages covered, as the source states them
source_url   the paper abstract, dataset card, or repository the number was read from
source_type  paper abstract | dataset card | README at commit <sha>
read_at      2026-09-05 for every row
note         the sentence or condition the number depends on

Twelve rows. Every number is quoted from an abstract, a dataset card, or a README at a named commit; none is a leaderboard score, because leaderboards are living tables and a CSV of them would be wrong within a week.

How to read the next score you see

Ask five questions. Which split: Full, Lite, Verified, Pro, Multimodal, or something else. Which harness, and did the paper vary it. Pass@1 or pass@k. Was the data public before the model's training cutoff. What did a pass cost. The score answers none of those on its own, and each one has moved a published number by double digits. That does not make the benchmark useless; it makes it a resolve rate under stated conditions, which is exactly what its authors built and what its evaluation guide says.

What I did not do: I did not run the harness, did not reproduce any score, and did not read any trajectory. Everything here is what the papers, the dataset card, the evaluation guide, and the repositories state on 2026-09-05.