KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference

A mixed-precision KV-cache quantization framework that searches layer-wise key/value precision pairs under hardware-friendly constraints. It shows why one KV precision setting rarely fits every layer or model, especially under long-context latency targets.

KVTuner helps explain why a single '2-bit cache' or '4-bit cache' label is too crude for GPU buying decisions.

It shows why one KV precision setting rarely fits every layer or model, especially under long-context latency targets. The useful part for GPU Hunter readers is not the abstract result alone; it is the hardware implication: whether a model fits, whether a runtime can use the format, or whether throughput is limited by memory movement instead of arithmetic.

KV cache quantization should be model-aware; uniform low-bit settings can waste quality or memory depending on the layer. For long-context work, KV cache behavior is often the constraint that shows up after the model weights already fit. Cache precision, eviction, reuse, and memory movement can change the practical value of the same GPU.