Morning Edition · Thursday, October 22, 2026Published at 2:23 AM EDT · New York

Two New Serving Results Cut Agent Latency: A Learned KV-Cache Retention Policy and an 8B Model That Scores Actions Instead of Generating Them

The KVLearn retention policy cuts time-to-first-token by up to 33 percent against Mooncake-style baselines in disaggregated serving, and Stanford and NVIDIA's open CLM-8B picks agent actions up to 9 times faster than its closest rival.

Disaggregated serving puts prefill and decode on separate node pools, so the network fabric sits between where the KV cache is computed and where it is used. A new arXiv paper argues that this setup breaks classical cache policies. A cache…