# Evaluation Design Becomes the Bottleneck

As capability claims outpace the validity of existing tests, benchmark construction turns into a contested research area and a gating input to enterprise procurement, so expect recurring purpose-built evaluations whose design, not whose leaderboard, is the news.

- Conviction: 100 / 100 (strengthening)
- 7-day move: +19
- Horizon: Medium term (3-9 months)
- Tracking since: 2026-09-18T00:00:00.000Z
- Last updated: 2026-10-12T14:07:13.413Z
- Canonical: https://polylog.news/ai/trends/evaluation-design-becomes-the-bottleneck
- Publisher: Polylog
- Affected regions: Global

## Recent score history

- 2026-10-11: 100
- 2026-10-12: 100

## Recent evidence

- [confirms] SWE-Duel benchmark pits coding agents against each other to resist training-data contamination (2026-10-12): The SWE-Duel benchmark has one coding agent ship a feature with a hidden bug and a rival agent find and fix it. Only test execution decides the score, with no model judge, which is meant to resist training-data contamination. The news here is the benchmark's design, a direct answer to doubts about whether static coding leaderboards are still valid.
- [confirms] Microsoft Ships Decision-1, a Qwen-Based Scorer That Returns Calibrated Option Probabilities at $0.042 per Million Tokens (2026-10-11): Critics say Microsoft's claimed 83.5% accuracy across 36 benchmarks rests on unnamed test sets, and that its 35x latency advantage over GPT-6 Sol shrinks to 1.4x on one raw timing comparison. The dispute is about how the evaluation was built, not the scores.

29 more evidence entries, the full score history, the conviction-driver timeline, and affected assets are for subscribers: https://polylog.news/pricing