Morning Edition · Thursday, October 8, 2026Published at 2:25 AM EDT · New York

Two new papers show standard metrics hiding real costs, from tokenizer penalties on non-Latin scripts to a diarizer that worsens while appearing to improve

Tokka-Bench compares seven tokenizers across 100 natural and 20 programming languages and finds that how a vocabulary is allocated matters more than its size.

Tokka-Bench is an open-source framework that scores tokenizers on five metrics: bytes per token, unique token coverage, subword fertility, word-split rate and vocabulary composition. It covers 100 natural languages in more than 30 scripts a…