OpenAI Withdraws Support for SWE-Bench Pro After Finding a Third of Tasks Defective

OpenAI has retracted its recommendation of SWE-Bench Pro as a leading coding benchmark, publishing an audit that found roughly a third of its tasks distort results. Using model-based investigator agents and five experienced human reviewers, the company's pipeline flagged 200 of the 731 public tasks (27.4%) as broken, while human reviewers identified issues in 249 tasks, or 34.1%.

The failure modes are specific. They include hidden requirements not stated in the prompt, contradictory instructions, hidden tests that are too strict, and incomplete grading criteria. All of these come from sourcing tasks automatically from repository feature changes, where the description, the merged code, and the tests do not cleanly align. OpenAI says the benchmark's usable signal saturates near a 70% noise ceiling even as reported scores rose from a 23.3% pass rate to 80.3% in eight months.

There is a self-interested angle worth naming. A leading lab that publishes a critique of a benchmark is also a lab shaping which numbers the field treats as authoritative. But the specific finding, that a widely cited evaluation mislabels correct solutions as failures on a third of its tasks, is a claim other groups can reproduce by re-reviewing the same public split. It points to a real problem, which is that coding scores are rising faster than the tools measuring them can be trusted.