Morning Edition · Friday, July 3, 2026Published at 7:03 AM EDT · New York

Inference-cost pressure produces both a serving paper and a prompt-stripping trend

A new method for compressing the key-value cache (the memory a model builds up as it generates text) targets the long chains of reasoning that make these models expensive to run, while developers manually cut their own token costs.

Reasoning models generate long chains of reasoning, and that verbosity accumulates a large key-value cache during decoding, which raises latency and limits throughput. A new paper, Kara, proposes sliding-window KV-cache compression tailored…