About a third of the best coding agents' failures are the benchmark's fault
I read every one of the 539 failures of the top five systems on SWE-bench Verified. Many of them are failures of the tests, not of the agents.
SWE-bench Verified is the number every coding-agent release leads with. The five best systems on its public leaderboard now resolve between 77 and 79 percent of its 500 tasks, and the fifth they miss has become shorthand for what frontier agents still cannot do.
I wanted to know how much of that fifth actually belongs to the agents. So instead of auditing tasks, I audited failures: every task attempt that one of the five systems got wrong, read next to the issue it was meant to fix, the reference solution, the hidden tests, the patch the agent submitted, and the log of the run that graded it.
There are 539 of them, across 166 tasks. Start with the cleanest cases. In 28, the evaluation harness broke a run that should have passed: a log parser that couldn’t match a test’s name, a test runner that turned a deprecation warning into 148 errors. Re-scored today, with the patches the systems actually submitted, they pass. Two more that looked like harness errors still fail, so they don’t count.
Next, the labels. Models from two different labs read every failure independently, without seeing each other’s answers. Both blame the benchmark for 94. That is the floor, a number even a skeptic of either model should accept.
The two mostly disagree about tests that ask for more than the issue does, so I settled it by running code. Each family wrote its own test from the issue text alone, and every disputed patch ran against both. With that evidence, 202 failures, 38 percent, are the benchmark’s fault.
Each square is one failure. Amber: caused by the benchmark, under each kind of evidence in turn. Outlined: too little evidence to judge. Tap the grid to replay, or a step to jump to it.
The obvious worry about a check like that is whether it depends on who wrote it. It barely does. The two families’ tests return the same verdict on 98 percent of the patches both can judge. Patches that only one family blamed on the test pass about 90 percent of the time, almost exactly as often as the ones both families blamed on it, while patches both families called genuine bugs pass only half the time. The second family’s own tests overturn most of its own “genuine bug” labels. Depending on whose tests you use, the share is 35 or 38 percent. Correcting for how often a lenient test lets a wrong patch through puts it at 36, and it stays above a third unless wrong patches pass those tests more than 70 percent of the time.
What the broken tests look like is almost mundane. Here is one that every one of the five systems fails.
A user can trigger a server crash… The issue happens as part of the validation logic run in form.is_valid, specifically, when calling the SelectDateWidget.value_from_datadict, where the user-controlled value is converted into a date without guarding against a possible OverflowError.from the issue text, Django ticket
tests = [
((None, "12", "1"), None),
(("2000", None, "1"), None),
(("2000", "12", None), None),
((str(sys.maxsize + 1), "12", "1"), "0-0-0"),
]
'9223372036854775808-12-1''0-0-0'The crash is fixed by all five systems. The test also requires the widget to return the string “0-0-0” for an out-of-range year, a choice the issue never mentions.
Every one of the five systems stopped the crash. None of them returned “0-0-0”, because nothing in the issue asks for it; the hidden test does. Variations of this repeat across repositories. In two pylint tasks, the test file imports a helper or a constant that only the reference fix creates, so any other correct fix fails before a single assertion runs. In a scikit-learn task, the issue suggests clipping a threshold to 1 and the test demands the opposite design, infinity. In an astropy task, all 242 tests pass, and the log parser, unable to match one test’s name, records the run as a failure anyway.
Count the failures the benchmark caused as resolved, and the leaderboard moves.
The leaderboard as published: a three-way tie at the top, all five within two points.
SWE-bench Verified resolve rate of the five highest-scoring public systems, with failures caused by the benchmark counted as resolved under each evidence tier. Grey: published score. Click a tier, or use the arrow keys.
Every system gains between six and ten points, and all five land between 85 and 89 percent. Read those as ceilings, not scores: I credited patches that were wrongly rejected but didn’t go looking for wrong patches that were accepted, and earlier audits show those exist too. The three-way tie at the top comes apart, with Sonar Foundation Agent ahead under every tier, but its lead is a couple of points, and the differences in how often each system ran into a broken task are within chance. What the figure does show is how little it takes. A few points of defect exposure is enough to reorder systems whose published scores sit within two points of each other.
The hardest tasks are where this matters most. Twenty-nine tasks in SWE-bench Verified have never been solved by any public submission.
Each tile is a task no public system has solved. Lit: tasks where all of the top five systems’ failures are benchmark defects.
A task nobody solves reads like a frontier challenge. On this benchmark, more than a third of the time, it is a broken test.
I think this changes how the leaderboard should be read. A one- or two-point gap between frontier systems on SWE-bench Verified is not a capability gap; it sits inside the noise that broken tasks alone produce. It matters even more for training. Anyone building reinforcement-learning environments out of issue and test pairs is using those hidden tests as the reward, and a failure the benchmark caused is a false-negative reward: the policy is penalised for a correct fix and pushed toward matching an unstated choice in someone else’s patch. The failures frontier policies still make are exactly the ones most likely to be like this, so the reward is most wrong where the signal is scarcest.
Some of it is cheap to catch. A test that imports a symbol neither the repository nor the issue mentions can be flagged without running anything, and a published score can be recomputed when a harness bug is fixed. The scores in this audit never were.