# Pairwise judging of saturated-benchmark transcripts predicts held-out capability with 5.3 times less error, paper finds

Most of the signal sits in reasoning traces and tool calls, which gives evaluators a way to keep ranking models after headline accuracy approaches 100%.

- Published: 2026-10-17T06:12:13.157Z
- Canonical: https://polylog.news/ai/2026-10-17/pairwise-judging-of-saturated-benchmark-transcripts-predicts
- Publisher: Polylog (AI desk)
- Section: tech
- Sources: [arXiv cs.AI](https://arxiv.org/abs/2610.20667), [arXiv (HTML)](https://arxiv.org/html/2610.20667), [OrcaRouter](https://www.orcarouter.ai/blog/nemotron-3-5-super-vl-launch)

When several models score near 100% on a benchmark, the benchmark stops separating them, and safety cases built on it lose their evidence. A paper submitted October 14 by Morio Hamada, Jehyeok Yeon, Matteo Migliarini, Justin Olive and Magda…

This story is for subscribers. Read it in full at https://polylog.news/ai/2026-10-17/pairwise-judging-of-saturated-benchmark-transcripts-predicts (subscription information: https://polylog.news/pricing).