# OpenAI Withdraws Support for SWE-Bench Pro After Finding a Third of Tasks Defective

Human reviewers flagged 249 of 731 public tasks (34.1%) as broken, and the benchmark's usable signal saturates near a 70% ceiling as models climbed from 23% to 80% in eight months.

- Published: 2026-07-09T05:32:14.424Z
- Canonical: https://polylog.news/ai/2026-07-09/openai-withdraws-support-for-swe-bench-pro-after-finding-a-t
- Publisher: Polylog (AI desk)
- Section: tech
- Sources: [OpenAI](https://openai.com/index/separating-signal-from-noise-coding-evaluations), [AlphaSignal](https://alphasignal.ai/news/openai-retracts-swe-bench-pro-after-finding-30-of-tasks-broken)

OpenAI has [retracted its recommendation of SWE-Bench Pro](https://openai.com/index/separating-signal-from-noise-coding-evaluations) as a leading coding benchmark, publishing an audit that found roughly a third of its tasks distort results. Using model-based investigator agents and five experienced human reviewers, the company's pipeline flagged 200 of the 731 public tasks (27.4%) as broken, while [human reviewers identified issues in 249 tasks, or 34.1%](https://alphasignal.ai/news/openai-retracts-swe-bench-pro-after-finding-30-of-tasks-broken).

The failure modes are specific. They include hidden requirements not stated in the prompt, contradictory instructions, hidden tests that are too strict, and incomplete grading criteria. All of these come from sourcing tasks automatically from repository feature changes, where the description, the merged code, and the tests do not cleanly align. OpenAI says the benchmark's usable signal saturates near a 70% noise ceiling even as reported scores rose from a 23.3% pass rate to 80.3% in eight months.

There is a self-interested angle worth naming. A leading lab that publishes a critique of a benchmark is also a lab shaping which numbers the field treats as authoritative. But the specific finding, that a widely cited evaluation mislabels correct solutions as failures on a third of its tasks, is a claim other groups can reproduce by re-reviewing the same public split. It points to a real problem, which is that coding scores are rising faster than the tools measuring them can be trusted.

## What this means

The exposed party is anyone making decisions off a single coding benchmark, including labs citing it in safety cases and buyers ranking models for procurement. When a third of an evaluation's tasks are broken, a reported six-point gap between two models can be noise, which means every claim of being state of the art on SWE-Bench Pro in recent marketing is open to question. The channel is credibility. Evaluations are how capability claims are settled, and a compromised benchmark degrades the entire comparison layer the industry depends on.

## What to watch

- Whether Scale AI and the SWE-Bench Pro maintainers rebut OpenAI's task-level findings or issue a cleaned dataset, which would show whether the defect rate is agreed or contested.
- Whether other labs stop citing SWE-Bench Pro in launch materials, a signal that the retraction has shifted the field's default benchmark rather than just OpenAI's.