Morning Edition · Monday, June 15, 2026Published at 3:00 AM EDT · New York

Study Finds LLM Judges Disagree With Themselves on Repeated Identical Runs

Re-running the same evaluation many times exposes run-to-run instability in the LLM-as-a-Judge method that underpins leaderboards and reward models.

A new paper, "The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation," studies what happens when the same model is asked to judge the same outputs repeatedly, in work posted to arXiv. The authors run repeated identical evalu…