Morning Edition · Tuesday, October 20, 2026Published at 2:12 AM EDT · New York

PolyGraph captures one CUDA graph for adaptive speculative decoding by writing a 28-byte shape plan before each replay

On a single H800, the prototype replaced 64 shape-specific graphs, matched warmed-baseline latency, and cut graph-build time before serving from 3,915 ms to 93 ms, though only on synthetic traces.

Adaptive speculative decoding changes the active batch size B and the draft length d from one decode step to the next. A CUDA graph replays a fixed launch configuration. That mismatch forces serving systems into one of three costly choices…