Morning Edition · Monday, August 31, 2026Published at 2:23 AM EDT · New York

Two New Papers Attack the Least Glamorous Bottlenecks in Serving Small Models

One reformulates the output projection as a vector search to relieve memory bandwidth pressure, the other quantizes the recurrent state that linear-attention models use instead of a growing cache.

Two arXiv papers posted today target costs that only show up once a model is actually being served at volume. The first addresses a bottleneck that grows worse as models get smaller. In a compact multilingual model, the output embedding mat…