Comparative Characterization of KV Cache Management Strategies for LLM Inference

An empirical comparison of KV cache management frameworks including vLLM, InfiniGen, and H2O across latency, throughput, memory, request rates, and model sizes. It gives buyers and builders a side-by-side view of when paging, offload, eviction, or sparse strategies actually help.

This is one of the best starter papers for turning long-context marketing claims into hardware and runtime tradeoffs.

It gives buyers and builders a side-by-side view of when paging, offload, eviction, or sparse strategies actually help. The useful part for GPU Hunter readers is not the abstract result alone; it is the hardware implication: whether a model fits, whether a runtime can use the format, or whether throughput is limited by memory movement instead of arithmetic.

The right KV cache strategy depends on workload shape; no one approach wins across every GPU, context length, and batch size. For long-context work, KV cache behavior is often the constraint that shows up after the model weights already fit. Cache precision, eviction, reuse, and memory movement can change the practical value of the same GPU.