OpenAI Retracts Its Endorsement of SWE-Bench Pro, Citing Broken Tasks

OpenAI formally retracted its earlier recommendation of SWE-Bench Pro as a leading coding evaluation, publishing an audit that found the benchmark saturating near a noise ceiling. Frontier pass rates on it climbed from 23.3 percent to about 80 percent in eight months, a rise OpenAI argues reflects the evaluation's flaws as much as model progress.

The audit examined the 731-task public split using model-based investigator agents and five experienced software engineers. The automated pipeline flagged 200 tasks, or 27.4 percent, as broken, while human reviewers found problems in 249 tasks, or 34.1 percent. The failures fall into four categories: tests that enforce implementation details the prompt never specified, prompts that omit requirements the hidden tests demand, low-coverage tests that pass incomplete fixes, and prompts that actively point a model toward the wrong behavior.

The timing is striking because every coding release this week, Grok 4.5 and Sonnet 5 included, cited SWE-Bench Pro numbers. OpenAI notes that evaluation flaws also distort safety cases and research priorities under its Preparedness Framework, tying the quality of benchmarks to governance rather than to leaderboard standing.