Morning Edition · Saturday, October 17, 2026Published at 2:12 AM EDT · New York

Pairwise judging of saturated-benchmark transcripts predicts held-out capability with 5.3 times less error, paper finds

Most of the signal sits in reasoning traces and tool calls, which gives evaluators a way to keep ranking models after headline accuracy approaches 100%.

When several models score near 100% on a benchmark, the benchmark stops separating them, and safety cases built on it lose their evidence. A paper submitted October 14 by Morio Hamada, Jehyeok Yeon, Matteo Migliarini, Justin Olive and Magda…