OpenAI audit finds about 30% of SWE-Bench Pro coding benchmark tasks are broken

Translation: other labs have just learned to benchmaxx SWE-Bench Pro better than OpenAI does.

AGI won't come from passing benchmarks, it has to handle unknown-unknowns too, and unless labs have something totally different in the lab, this is just marketing hype.

Depending on your field, everyone's building private benchmarks now. My own test is replicating a 1990s Super Soaker with Home Depot PVC parts and every frontier model completely fails at it, they just don't get spatial/mechanical reasoning.

LLMs seem to fail at spatial reasoning categorically. Might be because text data just doesn't teach that skill, or because real spatial reasoning needs true recurrence that autoregressive chain-of-thought can't substitute for.

My personal benchmark is build123d, trying to get models to build functional parts from a description alone. None of them perform well.

Didn't we already know all of SWE-Bench was flawed? Even the original authors admitted the limits and moved on already.

SWE-Bench Pro was literally created to replace SWE-Bench and fix exactly these problems, so this is kind of ironic.

What's actually SOTA for SWE benchmarks now? I've found DeepSWE pretty true to reality, and there's also FrontierBench and Cognition's FrontierCode which grades mergeability, i.e. whether a real dev would actually want to merge the output.

Interesting timing to release this right when SWE-1.7 and Grok 4.5 dropped, both way cheaper than GPT-5.5.

This really doesn't look like opportune timing to publish this right before a new model launch.

There's less than 800 tasks total in this benchmark, that's a week's work for a handful of engineers, which is basically what OpenAI did. Kudos for doing it, but embarrassing that neither the original authors nor anyone downstream checked first, and notably the LLM pipeline actually underestimated issues versus human engineers.

This reads like 'we did all the work to fix the benchmark then decided to just throw it out instead.' Why not patch the messy real-world tests into something fair instead of scrapping it?

Pointing out problems like overly-narrow hidden tests is way easier than actually fixing them, since you'd need tests that work for any valid implementation choice.

They say ~30% is broken but that leaves ~70% fine, which seems decent to me. Would be nice if they just published the CSV of broken instances or the cleaned subset instead of abandoning it entirely.

Aren't they basically concluding that real-world dev tasks are often incomplete or self-contradictory? That's just the world these tools have to operate in, I'm not that sympathetic to the complaint.

The real issue is the gap between task and verification: underspecified prompts need verifiers that handle any valid solution, and that's expensive and hard to build cheaply.

Calling 'underspecified prompts' a benchmark flaw is weak. Even interns handle ambiguous asks by asking clarifying questions instead of just barreling ahead, so are we saying frontier models can't even reach first-month-intern level?