Morning Edition · Friday, October 2, 2026Published at 2:31 AM EDT · New York

Reading an LLM Judge's First Token Inflates Position Bias by 42 Points, Paper Finds

Three Qwen3 judges failed to lead with a verdict token on 12% to 49% of pairs, and the cheap logit readout (scored from the model's raw output probabilities before any text is generated) distorts bias measurements while leaving accuracy nearly unchanged.

A new paper challenges a shortcut built into many evaluation harnesses. Reading a judge model's verdict from the logits of its first generated token requires no generation, costs almost nothing, and is exactly what constrained decoding and…