# OpenAI Retracts Its Endorsement of SWE-Bench Pro, Citing Broken Tasks

An audit of the 731-task public split flagged 27 percent of tasks as broken by an automated pipeline and 34 percent by human reviewers.

- Published: 2026-07-09T05:49:21.519Z
- Canonical: https://polylog.news/ai/2026-07-09/openai-retracts-its-endorsement-of-swe-bench-pro-citing-brok
- Publisher: Polylog (AI desk)
- Section: tech
- Sources: [OpenAI](https://openai.com/index/separating-signal-from-noise-coding-evaluations), [AlphaSignal](https://alphasignal.ai/news/openai-retracts-swe-bench-pro-after-finding-30-of-tasks-broken)

OpenAI [formally retracted](https://openai.com/index/separating-signal-from-noise-coding-evaluations) its earlier recommendation of SWE-Bench Pro as a leading coding evaluation, publishing an audit that found the benchmark saturating near a noise ceiling. Frontier pass rates on it climbed from 23.3 percent to about 80 percent in eight months, a rise OpenAI argues reflects the evaluation's flaws as much as model progress.

The audit examined the 731-task public split using model-based investigator agents and five experienced software engineers. The automated pipeline flagged 200 tasks, or 27.4 percent, as broken, while human reviewers found problems in 249 tasks, or [34.1 percent](https://alphasignal.ai/news/openai-retracts-swe-bench-pro-after-finding-30-of-tasks-broken). The failures fall into four categories: tests that enforce implementation details the prompt never specified, prompts that omit requirements the hidden tests demand, low-coverage tests that pass incomplete fixes, and prompts that actively point a model toward the wrong behavior.

The timing is striking because every coding release this week, Grok 4.5 and Sonnet 5 included, cited SWE-Bench Pro numbers. OpenAI notes that evaluation flaws also distort safety cases and research priorities under its Preparedness Framework, tying the quality of benchmarks to governance rather than to leaderboard standing.

## What this means

The measurement layer that the entire coding race is scored against is unreliable, and the lab most invested in coding capability is the one saying so. The mechanism is that programmatically sourced tasks conflate a merged commit's tests with the actual specification, so a model can be penalized for a correct fix or rewarded for an incomplete one. Everyone quoting a SWE-Bench Pro number this week, xAI and Anthropic included, is exposed, because the differences between models may fall inside the benchmark's own error rate. That means the honest ranking of frontier coding models is currently unknown.

## What to watch

- Whether SWE-Bench Pro's maintainers issue a cleaned split, and whether labs re-run their published numbers on it.
- Which benchmark the field converges on next, since a vacuum in trusted coding evals invites each lab to favor the test it happens to win.