SWE-bench explained: what a resolve rate measures and what it misses

A SWE-bench percentage is the most quoted number in agentic coding, and it is a narrower measurement than the way it is quoted. This piece explains exactly what a run does to produce the number, what the Verified subset fixed, and what five later papers found when they looked at the same score from other angles. The dataset ships every published number used here with its source. The numbers in how big a gap is real are ours: we computed them from the leaderboard's own per-task records with the script in the Code box.

What a run does

The benchmark comes from the 2023 paper by Carlos E. Jimenez, John Yang, and colleagues (arXiv:2310.06770). Its abstract describes "2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories," and reports that "The best-performing model, Claude 2, is able to solve a mere 1.96% of the issues." Each problem is an issue text, a repository at a base commit, and a hidden set of tests.

The SWE-bench evaluation guide states the procedure in one sentence: "SWE-bench evaluates models by applying their generated patches to real-world repositories and running the repository's tests to verify if the issue is resolved," inside "a containerized Docker environment to ensure consistent results across different platforms." A submission is a JSONL file, one object per instance:

{
  "instance_id": "repo_owner__repo_name-issue_number",
  "model_name_or_path": "your-model-name",
  "model_patch": "the patch content as a string"
}

The command is:

python -m swebench.harness.run_evaluation \
    --dataset_name princeton-nlp/SWE-bench_Lite \
    --predictions_path <path_to_predictions> \
    --max_workers 8 \
    --run_id my_evaluation_run

The definition of success is in the guide's results table: "Instances resolved: The patch made the required tests pass." Which tests are required comes from two columns in the dataset. The Verified dataset card defines FAIL_TO_PASS as "A json list of strings that represent the set of tests resolved by the PR and tied to the issue resolution," and PASS_TO_PASS as "A json list of strings that represent tests that should pass before and after the PR application." Resolved means the first set flips to passing and the second set stays passing. The card also defines patch as "The gold patch, the patch generated by the PR (minus test-related code), that resolved the issue," and test_patch as "A test-file patch that was contributed by the solution PR."

So the number is: the fraction of instances where the agent's patch, applied at base_commit, makes the maintainers' own tests for that issue pass without breaking the others. It is a pass@1 unless a submission says otherwise; the harness scores one patch per instance.

What Verified changed

The repository README at commit 02e7a74ffd0b (2026-09-02; 5,781 stars, MIT, read 2026-09-05) records the milestones: Docker-based evaluation in June 2024, SWE-bench Verified in August 2024 as "A subset of 500 problems that real software engineers have confirmed are solvable," and, on 2026-09-01, SWE-bench Multimodal v2 with 480 tasks. The Verified dataset card describes its 500 rows as "a subset of 500 samples from the SWE-bench test set, which have been human-validated for quality." The point of validation was that some original issues were underspecified or had tests that a correct fix could still fail; the subset removed those. Most quoted scores since late 2024 are on Verified, and a score on Verified is not comparable with one on the full test set or on Lite.

The leaderboard site lists Verified, Multimodal, Multilingual, Lite, and Full tabs (read 2026-09-05; we do not quote scores from it, because they change and this piece would go stale). What the leaderboard requires is documented in the experiments repository, which "contains records of submissions to the SWE-bench leaderboard." A submission needs all_preds.jsonl, per-instance logs with the patch, evaluation report, and test output, and "Human-readable" reasoning traces that reflect "the intermediate steps your system took that led to the final solution." Verification is by re-derivation: "swebench submit verify re-derives every verdict from test output," with "no Docker, no re-execution." That means the leaderboard checks that your logs support your score; it does not rerun your agent.

What the score does not carry

The harness. Sydney Lewis's paper (arXiv:2608.26218) held the model and tasks fixed and changed only the harness's context management: the control showed the full conversation chronologically, the treatment kept the same records but "progressively condensed older tool outputs as context limits approached." On 169 Verified tasks with a 20,480-token window and 480-second attempts, the fail-to-pass fraction moved from 28% to 49%, and complete solutions from 43 to 72. Same model, same tasks, 21 points. A leaderboard row names a system, and a system is a model plus a harness, so a row cannot be read as a model score. This is also why the SWE-agent paper (arXiv:2405.15793) mattered: its 12.5% pass@1 on the full set was an interface result as much as a model result, as discussed in the SWE-agent reading.

Whether the pass was earned. AgentLens (arXiv:2605.12925) analyzed 2,614 OpenHands trajectories from eight model backends on 60 Verified tasks and found that "Among passing trajectories in this subset, 10.7% exhibit behavior we call a Lucky Pass," meaning regression cycles, blind retries, missing verification, or temporally disordered steps. The per-model range was 0.5% to 23.2%, and "some models move by as many as five rank positions when ranked by quality score instead of pass rate." The score counts the patch, not the path.

Contamination. Prathifkumar, Mathews, and Nagappan (arXiv:2512.10218) tested file localization and found models performed "3 times better" on Verified than on alternative benchmarks and were "6 times" better at naming the edited files without context, which they read as evidence that Verified may have been seen in training. A score on public data has a memorization component that is not separable from the number.

The prompt. RealSWE (arXiv:2608.27831) compared real user requests with benchmark problem statements and found that "88% of real prompts but just 7% of benchmark problems" contain only a bare problem statement, and "87% of real prompts are casually written whereas 94% of benchmark problems are formal." Rewriting 381 task families in realistic styles cut resolution by 6.4 percentage points on average across seven models. A benchmark issue is a better-specified task than the one you will type, a point spec-first prompting turns into practice.

Difficulty ceiling. SWE-Bench Pro (arXiv:2509.16941) was built because Verified saturated; it has "1,865 problems sourced from a diverse set of 41 actively maintained repositories." Its abstract reports no resolve rate, so we report none.

Cost. Nothing in the resolve rate says how many tokens, dollars, or minutes a pass took. The FailFast paper (arXiv:2608.03222) shows how much slack is there: an early-stop monitor saved "14.6%-20.4% of execution tokens at a target 5% false-positive rate" on Verified with no change to the task. Two systems with the same score can differ by that much in cost, and the method in what one task costs is the missing axis.

How big a gap is real: what the per-task records add

Everything above is read from papers and documentation. This section adds numbers we computed from the per-task records behind the task-by-task chapter: the eleven runs that held the agent fixed (mini-swe-agent 2.0.0, one attempt per task, each record agreeing with its own entry's stated score) and all 175 Verified entries in the SWE-bench/experiments repository, at commit 40f164d5b8f1, that record which tasks they resolved. The responses were saved on 8 October 2026. The script re-reads them, re-derives no verdict and runs no agent.

One task is 0.2 points, and a score is wider than it looks. A Verified score counts 500 tasks, so it moves in steps of 0.2 points, and it is exact for those 500. Read as a sample of issues like them, which is how scores get used, it is wider: the 95% Wilson interval around each of the eleven scores is 7.4 to 8.7 points wide, so the best run's 76.8% reads as 72.9% to 80.3%.

Gaps under about three points did not separate. Two runs on the same tasks differ only on the tasks they split, the ones one run resolved and the other did not. The exact McNemar test asks whether that split is more lopsided than a fair coin would make it. Of the 55 pairs among the eleven runs, 33 were separable at p < 0.05, and 24 after Holm's correction for running 55 tests. In these records the line was sharp: no gap of 15 tasks (3.0 points) or less was separable, and every gap of 17 tasks (3.4 points) or more was. A small gap can hide a lot of disagreement. Claude 4.5 Opus (high) and Gemini 3 Flash (high), five tasks apart, split 53 tasks, and every pair of runs split between 49 and 129.

The top four are a tie, and so is the top of the board. We resampled the 500 tasks with replacement 10,000 times and ranked the eleven runs again each time. Claude 4.5 Opus (high) came first in 53.9% of the resamples, Gemini 3 Flash (high) in 17.0%, MiniMax M2.5 (high) in 16.2% and Claude 4.6 Opus in 12.8%; the other seven together came first in 7.5 of the 10,000. The same pairwise test across the whole board gives the same picture. The two entries tied at the top with 396 resolved split 36 tasks, 18 to 18, and neither can be separated from the 7 entries that resolved 382 to 394, among them the bash-only Claude 4.5 Opus (high) run at 384. All 166 other entries, the highest of them at 379, can be. On these records, the top of the Verified board is a group of nine, not a ranking.

One repository is 46% of the score. Django supplies 231 of the 500 tasks (46.2%); the other 11 repositories supply 269. Every one of the eleven runs resolved a larger share of the Django tasks, and the difficulty labels do not explain it away. With the non-Django tasks reweighted to Django's mix of time labels, Django's rate was higher by 3.5 to 10.9 points in all 11 runs. Pooled over the runs, Django's 15-minute-to-an-hour tasks were resolved 73.6% of the time, against 63.6% elsewhere, and its one-to-four-hour tasks 44.6%, against 20.5%. The order moves too. On the 269 non-Django tasks, Claude 4.6 Opus (74.0%) is second and Gemini 3 Flash (high) (72.1%) fourth, where on all 500 they were fourth and joint second. The records cannot say why Django tasks go better. Memorization, as in the contamination paper above, is one candidate; better-written issues or tests are others. What they do say is that a Verified score is close to half a Django score.

What this leaves out. Each run made one attempt per task, so these tests treat the choice of tasks as the only source of chance. A second attempt by the same model could flip some tasks, and the records cannot say how many; any such flips would make the real intervals wider, not narrower. The eleven runs share one agent, so the comparisons are between models under that agent; the board-wide comparison mixes agents, and a gap there is a gap between systems.

Run it

The script in the Code box is standard-library Python. It reads the same folder of saved responses as the task-by-task script (--cache), or fetches them live from the pinned commit without it, and the resampling uses a fixed seed, so a rerun prints the same numbers. This is the run behind every number in this section, verbatim:

$ python swe-bench-explained-what-the-numbers-measure.py --cache saved
SWE-bench/experiments at 40f164d5b8f1: 175 Verified entries with a per-task record; 500 tasks
mini-swe-agent 2.0.0 entries: 13; runs used (a full record that agrees with its own metadata): 11

1. One score on 500 tasks, with a 95% Wilson interval
  run                        resolved   score   95% interval     width
  Claude 4.5 Opus (high)        384    76.8%   72.9% to 80.3%   7.4 pts
  Gemini 3 Flash (high)         379    75.8%   71.9% to 79.3%   7.5 pts
  MiniMax M2.5 (high)           379    75.8%   71.9% to 79.3%   7.5 pts
  Claude 4.6 Opus               378    75.6%   71.6% to 79.2%   7.5 pts
  GLM 5 (high)                  364    72.8%   68.7% to 76.5%   7.8 pts
  GPT 5.2 (high)                364    72.8%   68.7% to 76.5%   7.8 pts
  Claude 4.5 Sonnet (high)      357    71.4%   67.3% to 75.2%   7.9 pts
  Kimi K2.5 (high)              354    70.8%   66.7% to 74.6%   7.9 pts
  DeepSeek V3.2 (high)          350    70.0%   65.8% to 73.9%   8.0 pts
  Claude 4.5 Haiku (high)       333    66.6%   62.4% to 70.6%   8.2 pts
  GPT 5 mini                    281    56.2%   51.8% to 60.5%   8.7 pts
  interval width: 7.4 to 8.7 points; one task = 0.2 points

2. Every pair of runs, exact two-sided McNemar test on the tasks they split
  pairs: 55; separable at p < 0.05: 33; after Holm's correction for 55 tests: 24
  largest gap not separable: 15 tasks (3.0 points); smallest gap separable: 17 tasks (3.4 points)
  split tasks per pair: 49 to 129
  run A                      run B                      gap  A only  B only       p
  GLM 5 (high)               GPT 5.2 (high)               0      29      29   1.000
  Gemini 3 Flash (high)      MiniMax M2.5 (high)          0      28      28   1.000
  Gemini 3 Flash (high)      Claude 4.6 Opus              1      33      32   1.000
  MiniMax M2.5 (high)        Claude 4.6 Opus              1      33      32   1.000
  Claude 4.5 Sonnet (high)   Kimi K2.5 (high)             3      28      25   0.784
  Kimi K2.5 (high)           DeepSeek V3.2 (high)         4      33      29   0.704
  Claude 4.5 Opus (high)     Gemini 3 Flash (high)        5      29      24   0.583
  Claude 4.5 Opus (high)     MiniMax M2.5 (high)          5      30      25   0.590
  Claude 4.5 Opus (high)     Claude 4.6 Opus              6      30      24   0.497
  Claude 4.5 Sonnet (high)   DeepSeek V3.2 (high)         7      31      24   0.419
  GLM 5 (high)               Claude 4.5 Sonnet (high)     7      34      27   0.443
  GPT 5.2 (high)             Claude 4.5 Sonnet (high)     7      34      27   0.443
  GLM 5 (high)               Kimi K2.5 (high)            10      38      28   0.268
  GPT 5.2 (high)             Kimi K2.5 (high)            10      42      32   0.295
  Claude 4.6 Opus            GLM 5 (high)                14      43      29   0.125
  Claude 4.6 Opus            GPT 5.2 (high)              14      43      29   0.125
  GLM 5 (high)               DeepSeek V3.2 (high)        14      37      23   0.092
  GPT 5.2 (high)             DeepSeek V3.2 (high)        14      35      21   0.081
  Gemini 3 Flash (high)      GLM 5 (high)                15      36      21   0.063
  Gemini 3 Flash (high)      GPT 5.2 (high)              15      37      22   0.067
  MiniMax M2.5 (high)        GLM 5 (high)                15      41      26   0.086
  MiniMax M2.5 (high)        GPT 5.2 (high)              15      43      28   0.096
  DeepSeek V3.2 (high)       Claude 4.5 Haiku (high)     17      38      21   0.036

3. Ranked first when the 500 tasks are resampled (10,000 resamples, seed 20261105, ties split)
  Claude 4.5 Opus (high)      53.9%    5,394.7 of 10,000
  Gemini 3 Flash (high)       17.0%    1,703.2 of 10,000
  MiniMax M2.5 (high)         16.2%    1,618.0 of 10,000
  Claude 4.6 Opus             12.8%    1,276.7 of 10,000
  GLM 5 (high)                 0.0%        4.7 of 10,000
  GPT 5.2 (high)               0.0%        2.8 of 10,000
  Claude 4.5 Sonnet (high)     0.0%        0.0 of 10,000
  Kimi K2.5 (high)             0.0%        0.0 of 10,000
  DeepSeek V3.2 (high)         0.0%        0.0 of 10,000
  Claude 4.5 Haiku (high)      0.0%        0.0 of 10,000
  GPT 5 mini                   0.0%        0.0 of 10,000

4. All entries with a per-task record: who cannot be separated from the top score
  most resolved: 396 (20251205_sonar-foundation-agent_claude-opus-4-5, 20251215_livesweagent_claude-opus-4-5)
  the first two split 36 tasks, 18 to 18
  not separable from the top at p < 0.05: 7 entries, 382 to 394 resolved
    394  20250928_trae_doubao_seed_code
    388  20251127_openhands_claude-opus-4-5
    387  20251120_livesweagent_gemini-3-pro-preview
    384  20250804_epam-ai-run-claude-4-sonnet
    384  20250902_atlassian-rovo-dev
    384  20260217_mini-v2.0.0_claude-4-5-opus-high
    382  20250819_ACoder
  separable from the top: 166 entries, the highest of them 379 resolved

5. One repository's weight in the score
  django/django: 231 of 500 tasks (46.2%); the other 11 repositories: 269
  <15 min fix      django/django:  92 tasks, all runs  85.2%   others: 102 tasks, all runs  84.2%
  15 min - 1 hour  django/django: 117 tasks, all runs  73.6%   others: 144 tasks, all runs  63.6%
  1-4 hours        django/django:  22 tasks, all runs  44.6%   others:  20 tasks, all runs  20.5%
  >4 hours         django/django:   0 tasks, all runs      -   others:   3 tasks, all runs  27.3%
  run                         score  django  others  others at django's label mix  difference
  Claude 4.5 Opus (high)      76.8%   78.8%   75.1%                        74.8%   +4.0 pts
  Gemini 3 Flash (high)       75.8%   80.1%   72.1%                        71.8%   +8.3 pts
  MiniMax M2.5 (high)         75.8%   79.2%   72.9%                        72.7%   +6.6 pts
  Claude 4.6 Opus             75.6%   77.5%   74.0%                        74.0%   +3.5 pts
  GLM 5 (high)                72.8%   78.4%   68.0%                        68.2%  +10.1 pts
  GPT 5.2 (high)              72.8%   78.4%   68.0%                        68.4%  +10.0 pts
  Claude 4.5 Sonnet (high)    71.4%   74.9%   68.4%                        68.3%   +6.6 pts
  Kimi K2.5 (high)            70.8%   76.6%   65.8%                        65.7%  +10.9 pts
  DeepSeek V3.2 (high)        70.0%   75.3%   65.4%                        65.1%  +10.3 pts
  Claude 4.5 Haiku (high)     66.6%   69.3%   64.3%                        63.9%   +5.3 pts
  GPT 5 mini                  56.2%   61.5%   51.7%                        51.7%   +9.8 pts
  runs with a higher rate on django/django than on the others at its label mix: 11 of 11 (+3.5 to +10.9 pts)
  order on the other 269 tasks: Claude 4.5 Opus (high) 75.1%, Claude 4.6 Opus 74.0%, MiniMax M2.5 (high) 72.9%, Gemini 3 Flash (high) 72.1%, ...

Two numbers, two years apart

The SWE-agent paper reported 12.5% pass@1 on the full 2,294-instance test set in 2024. The mini-swe-agent README at commit 04d809ceab9d claims "Scores >74% on the SWE-bench verified benchmark" in 2026. Those are different benchmarks, different sizes, different harness philosophies (fifteen tool bundles versus bash only), and the second is a README claim we did not reproduce. The change is real and large, and the comparison is still not a comparison. The dataset records both with their source_type, so the paper number and the README number are never mistaken for the same kind of evidence.

The dataset

_README for swe-bench-explained-what-the-numbers-measure.csv
name         the benchmark, result, or finding
kind         benchmark | benchmark subset | result | result claim | finding
size         the number as published (count, percent, or multiplier)
unit         what the number counts; percent rows say "percent resolved" or "percent fail-to-pass"
languages    programming languages covered, as the source states them
source_url   the paper abstract, dataset card, or repository the number was read from
source_type  paper abstract | dataset card | README at commit <sha>
read_at      2026-09-05 for every row
note         the sentence or condition the number depends on

Twelve rows. Every number is quoted from an abstract, a dataset card, or a README at a named commit; none is a leaderboard score, because leaderboards are living tables and a CSV of them would be wrong within a week.

How to read the next score you see

Ask five questions. Which split: Full, Lite, Verified, Pro, Multimodal, or something else. Which harness, and did the paper vary it. Pass@1 or pass@k. Was the data public before the model's training cutoff. What did a pass cost. The score answers none of those on its own, and each one has moved a published number by double digits. That does not make the benchmark useless; it makes it a resolve rate under stated conditions, which is exactly what its authors built and what its evaluation guide says. Before you read an order into two scores, check the gap as well: in the per-task records, no gap of three points or less between two runs on the same 500 tasks could be told apart from chance.

What we did not do: we did not run the harness, did not reproduce any score, and did not read any trajectory. The gap section re-reads the per-task records that submitters committed to the leaderboard, saved on 2026-10-08, and re-derives none of them. Everything else here is what the papers, the dataset card, the evaluation guide, and the repositories state on 2026-09-05.

Update, 5 November 2026: added how big a gap is real, computed from the per-task records of the eleven mini-swe-agent 2.0.0 runs and of all 175 Verified entries that have one: the interval around a score, which pairs of runs can be told apart, how often each run ranks first when the tasks are resampled, which entries tie the top of the board, and how much of a score comes from Django. The script that computes them now ships in the Code box. The description and the opening say so.