Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs

An adaptive KV-cache quantization approach for mobile, embedded, and edge LLM inference where memory bandwidth and cache growth dominate. On-device inference cannot afford fixed precision everywhere; wasting bits directly reduces usable context and throughput.

This paper is useful for budget and mobile-class hardware because every unnecessary cache bit reduces context or throughput.

On-device inference cannot afford fixed precision everywhere; wasting bits directly reduces usable context and throughput. The useful part for GPU Hunter readers is not the abstract result alone; it is the hardware implication: whether a model fits, whether a runtime can use the format, or whether throughput is limited by memory movement instead of arithmetic.

Small GPUs, laptops, and edge boxes need cache precision policies that spend memory only where the model is sensitive. For long-context work, KV cache behavior is often the constraint that shows up after the model weights already fit. Cache precision, eviction, reuse, and memory movement can change the practical value of the same GPU.