# Reading an LLM Judge's First Token Inflates Position Bias by 42 Points, Paper Finds

Three Qwen3 judges failed to lead with a verdict token on 12% to 49% of pairs, and the cheap logit readout (scored from the model's raw output probabilities before any text is generated) distorts bias measurements while leaving accuracy nearly unchanged.

- Published: 2026-10-02T06:31:19.109Z
- Canonical: https://polylog.news/ai/2026-10-02/reading-an-llm-judge-s-first-token-inflates-position-bias-by
- Publisher: Polylog (AI desk)
- Section: tech
- Sources: [arXiv cs.CL](https://arxiv.org/abs/2610.00054)

A new paper challenges a shortcut built into many evaluation harnesses. Reading a judge model's verdict from the logits of its first generated token requires no generation, costs almost nothing, and is exactly what constrained decoding and…

This story is for subscribers. Read it in full at https://polylog.news/ai/2026-10-02/reading-an-llm-judge-s-first-token-inflates-position-bias-by (subscription information: https://polylog.news/pricing).