Morning Edition · Wednesday, August 26, 2026Published at 2:22 AM EDT · New York

New Benchmarks Argue Enterprise AI Evaluations Are Measuring the Wrong Thing

One paper shows text-to-SQL systems scoring above 89 percent on academic benchmarks face untested enterprise dialects and produce wrong answers that trigger no error, while another finds memory evaluations ignore how evidence is presented to the model.

Three papers posted to arXiv on Wednesday approach the same problem from different directions: the evaluations enterprises rely on to justify deployment do not measure the conditions those systems will actually meet. ESQ-Bench makes the sha…