Morning Edition · Thursday, July 9, 2026Published at 1:49 AM EDT · New York

OpenAI Retracts Its Endorsement of SWE-Bench Pro, Citing Broken Tasks

An audit of the 731-task public split flagged 27 percent of tasks as broken by an automated pipeline and 34 percent by human reviewers.

Save

OpenAI Retracts Its Endorsement of SWE-Bench Pro, Citing Broken Tasks

OpenAI formally retracted its earlier recommendation of SWE-Bench Pro as a leading coding evaluation, publishing an audit that found the benchmark saturating near a noise ceiling. Frontier pass rates on it climbed from 23.3 percent to about 80 percent in eight months, a rise OpenAI argues reflects the evaluation's flaws as much as model progress.

The audit examined the 731-task public split using model-based investigator agents and five experienced software engineers. The automated pipeline flagged 200 tasks, or 27.4 percent, as broken, while human reviewers found problems in 249 tasks, or 34.1 percent. The failures fall into four categories: tests that enforce implementation details the prompt never specified, prompts that omit requirements the hidden tests demand, low-coverage tests that pass incomplete fixes, and prompts that actively point a model toward the wrong behavior.

The timing is striking because every coding release this week, Grok 4.5 and Sonnet 5 included, cited SWE-Bench Pro numbers. OpenAI notes that evaluation flaws also distort safety cases and research priorities under its Preparedness Framework, tying the quality of benchmarks to governance rather than to leaderboard standing.

Veracity: Corroborated

80/100

If true, who benefits: OpenAI, which gains by discrediting a benchmark that its rivals Grok 4.5 and Sonnet 5 cite prominently, reframing the coding race around measurement it controls the narrative on.
The nuance: The 27 and 34 percent broken-task findings are credible and specific, but the audit is OpenAI's own, timed against competitors' launches, and OpenAI has a direct interest in which evaluation the field adopts next.

An open-source-intelligence read of how likely this story is true with its real nuance, not a judgment of any outlet. It assesses the claim, weighing independent and adversarial reporting. How we label confidence.

What this means

The measurement layer that the entire coding race is scored against is unreliable, and the lab most invested in coding capability is the one saying so. The mechanism is that programmatically sourced tasks conflate a merged commit's tests with the actual specification, so a model can be penalized for a correct fix or rewarded for an incomplete one. Everyone quoting a SWE-Bench Pro number this week, xAI and Anthropic included, is exposed, because the differences between models may fall inside the benchmark's own error rate. That means the honest ranking of frontier coding models is currently unknown.

What to watch

Whether SWE-Bench Pro's maintainers issue a cleaned split, and whether labs re-run their published numbers on it.
Which benchmark the field converges on next, since a vacuum in trusted coding evals invites each lab to favor the test it happens to win.

Observations to monitor, not financial advice.

2 sources

Synthesized from: OpenAI · AlphaSignal

Part of a tracked trend

Oversight and Evaluation Lag Accelerating AI Capabilities

Over the next 3-6 months, evidence mounts that governance, evaluation, and agent-safety methods are failing to keep pace with capability growth, driving investment in interpretability, agent-manipulation benchmarks, and institutional-reform proposals.