SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving

A practical INT4 KV-cache method designed around paged layouts, regular memory access, and fused attention execution. Many KV compression papers look good offline but fail when the serving engine needs predictable memory access and fast kernels.

SAW-INT4 sharpens the buying question from 'how much VRAM?' to 'does my runtime have a cache format that can use this GPU efficiently?'

Many KV compression papers look good offline but fail when the serving engine needs predictable memory access and fast kernels. The useful part for GPU Hunter readers is not the abstract result alone; it is the hardware implication: whether a model fits, whether a runtime can use the format, or whether throughput is limited by memory movement instead of arithmetic.

For vLLM-style serving, deployability matters as much as compression ratio; the cache format has to fit the kernel path. For long-context work, KV cache behavior is often the constraint that shows up after the model weights already fit. Cache precision, eviction, reuse, and memory movement can change the practical value of the same GPU.