# Two papers argue that benchmark scores on interactive LLM tasks measure the evaluation design more than the model

One shows that overall accuracy can invert conclusions when comparing responses to different follow-up prompts, and the other finds that scored success in AI math tutoring shifts with the grading criteria chosen.

- Published: 2026-10-26T06:26:37.653Z
- Canonical: https://polylog.news/ai/2026-10-26/two-papers-argue-that-benchmark-scores-on-interactive-llm-ta
- Publisher: Polylog (AI desk)
- Section: tech
- Sources: [arXiv cs.CL](https://arxiv.org/abs/2610.31678), [arXiv cs.CL (MisconceptDialogue)](https://arxiv.org/abs/2610.31679), [arXiv cs.CL (structured generation)](https://arxiv.org/abs/2610.31681)

Two papers posted to arXiv on October 26 attack the same problem from different directions: when a benchmark measures an interactive exchange rather than a single answer, the measured result depends heavily on choices made by whoever design…

This story is for subscribers. Read it in full at https://polylog.news/ai/2026-10-26/two-papers-argue-that-benchmark-scores-on-interactive-llm-ta (subscription information: https://polylog.news/pricing).