Speculative Decoding: Performance or Illusion?

A production-grade vLLM study of speculative decoding variants across workloads, model scales, and batch sizes. Speculative decoding can look excellent in small research demos while underperforming under realistic batching and serving pressure.

This paper prevents a common buying mistake: assuming a runtime feature will speed up every workload on every GPU.

Speculative decoding can look excellent in small research demos while underperforming under realistic batching and serving pressure. The useful part for GPU Hunter readers is not the abstract result alone; it is the hardware implication: whether a model fits, whether a runtime can use the format, or whether throughput is limited by memory movement instead of arithmetic.

Use speculative decoding carefully: the speedup depends on workload, draft method, batch size, and engine implementation. For shared inference, the important question is not only how fast one prompt runs. Batching, scheduling, cache placement, and request mix decide whether a GPU behaves like a reliable service.