XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerialization

A cache-rematerialization approach that trades extra computation for lower KV cache memory traffic and storage pressure. Modern GPUs often have more compute growth than memory bandwidth growth, making recomputation attractive in the right regime.

This paper gives a rigorous reason why more TFLOPS may not improve long-context throughput as much as better memory behavior.

Modern GPUs often have more compute growth than memory bandwidth growth, making recomputation attractive in the right regime. The useful part for GPU Hunter readers is not the abstract result alone; it is the hardware implication: whether a model fits, whether a runtime can use the format, or whether throughput is limited by memory movement instead of arithmetic.

A faster GPU is not always the answer; sometimes spending compute to reduce memory movement is the better trade. For long-context work, KV cache behavior is often the constraint that shows up after the model weights already fit. Cache precision, eviction, reuse, and memory movement can change the practical value of the same GPU.