▲Evaluation Design Becomes the Bottleneck

Score history

Daily conviction score, 0 to 100. Higher means the thesis is more strongly corroborated.

Now 100

Showing the last few days. Unlock full score history.

Why the conviction moved

• Oct 12
  Strengthened +3

  The SWE-Duel benchmark has one coding agent ship a feature with a hidden bug and a rival agent find and fix it. Only test execution decides the score, with no model judge, which is meant to resist training-data contamination. The news here is the benchmark's design, a direct answer to doubts about whether static coding leaderboards are still valid.

• Oct 11
  Strengthened +2

  Critics say Microsoft's claimed 83.5% accuracy across 36 benchmarks rests on unnamed test sets, and that its 35x latency advantage over GPT-6 Sol shrinks to 1.4x on one raw timing comparison. The dispute is about how the evaluation was built, not the scores.

• Oct 11
  Strengthened +3

  Scale's Humanity's Sixth Sense benchmark puts the best model at 53.6% against 93.1% for humans. Its analysis of 8,573 failures traces 94% to perception or inference errors and 5% to logic, so the benchmark's value is in diagnosing where models fail, not in its leaderboard.

Showing the last 2 days. Unlock the full record.

Source trail

• Supporting · October 12, 2026

  SWE-Duel benchmark pits coding agents against each other to resist training-data contamination

  The SWE-Duel benchmark has one coding agent ship a feature with a hidden bug and a rival agent find and fix it. Only test execution decides the score, with no model judge, which is meant to resist training-data contamination. The news here is the benchmark's design, a direct answer to doubts about whether static coding leaderboards are still valid.

  arXiv cs.CL

• Supporting · October 11, 2026

  Microsoft Ships Decision-1, a Qwen-Based Scorer That Returns Calibrated Option Probabilities at $0.042 per Million Tokens

  Critics say Microsoft's claimed 83.5% accuracy across 36 benchmarks rests on unnamed test sets, and that its 35x latency advantage over GPT-6 Sol shrinks to 1.4x on one raw timing comparison. The dispute is about how the evaluation was built, not the scores.

  AI ML Big Data