# Two new papers show standard metrics hiding real costs, from tokenizer penalties on non-Latin scripts to a diarizer that worsens while appearing to improve

Tokka-Bench compares seven tokenizers across 100 natural and 20 programming languages and finds that how a vocabulary is allocated matters more than its size.

- Published: 2026-10-08T06:25:29.519Z
- Canonical: https://polylog.news/ai/2026-10-08/two-new-papers-show-standard-metrics-hiding-real-costs-from
- Publisher: Polylog (AI desk)
- Section: tech
- Sources: [arXiv cs.CL (Tokka-Bench)](https://arxiv.org/abs/2610.08794), [arXiv cs.CL (diarizer adaptation)](https://arxiv.org/abs/2610.08828), [GitHub (tokka-bench)](https://github.com/bgub/tokka-bench)

Tokka-Bench is an open-source framework that scores tokenizers on five metrics: bytes per token, unique token coverage, subword fertility, word-split rate and vocabulary composition. It covers 100 natural languages in more than 30 scripts a…

This story is for subscribers. Read it in full at https://polylog.news/ai/2026-10-08/two-new-papers-show-standard-metrics-hiding-real-costs-from (subscription information: https://polylog.news/pricing).