GPU inference optimization papers on memory bandwidth, FlashAttention, NF4 dequantization, kernel maturity, batching, and LLM bottlenecks.
LLM inference performance is often limited by memory movement and kernel implementation, not just advertised FLOPS. This is why two GPUs with similar paper specs can behave differently in llama.cpp, vLLM, TensorRT-LLM, MLX, or ROCm-backed runtimes.
These papers explain the core GPU systems ideas behind GPU Hunter rankings: HBM traffic, attention tiling, dequantization kernels, decode-specific kernels, and benchmark methods that expose real bottlenecks.
Read these first if you want the fastest path from research to a hardware decision.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Christopher Re
An IO-aware exact attention algorithm that reduces HBM traffic by tiling attention through SRAM.
GPU Hunter takeaway: Long context needs memory-efficient kernels; VRAM capacity alone does not guarantee usable speed.
Pol G. Recasens, Ferran Agullo, Yue Zhu, Chen Wang, Eun Kyung Lee, Olivier Tardieu
A GPU-level analysis showing large-batch LLM inference can remain DRAM-bandwidth bound even when conventional explanations call it compute-bound.
GPU Hunter takeaway: Batch-size scaling should be benchmarked against memory bandwidth behavior, not inferred from TFLOPS alone.
Shanli Xing, Yiyan Zhai, Alexander Jiang, Yixin Dong, Yong Wu, Zihao Ye
A benchmark framework connecting GPU kernel definitions, workloads, implementations, and evaluations for real inference systems.
GPU Hunter takeaway: Kernel maturity is a hardware feature in practice; the same GPU can behave very differently across inference backends.
These papers go deeper into kernels, cache policy, low-bit formats, or serving tradeoffs.
Xiangbo Qi, Chaoyi Jiang, Murali Annavaram
A shared-memory kernel optimization for accelerating NF4 dequantization on NVIDIA GPUs during quantized LLM inference.
GPU Hunter takeaway: Quantized model size is not enough; GPU Hunter benchmarks should care whether the backend has fast unpacking and dequant kernels.
Elias Frantar, Roberto L. Castro, Jiale Chen, Dan Alistarh
A mixed-precision kernel design for keeping 4-bit weight inference fast across useful batch sizes.
GPU Hunter takeaway: A quantized model is only as fast as the kernels that can consume its packed weights efficiently.
Ganesh Bikshandi, Jay Shah
A detailed look at implementing FlashAttention-2 on Hopper with CUTLASS, TMA, WGMMA, and fused CUDA kernels.
GPU Hunter takeaway: Architecture-specific kernel support is a real buying consideration for workstation and datacenter GPUs.
The complete paper set for this topic, with source links and GPU Hunter takeaways.
Pol G. Recasens, Ferran Agullo, Yue Zhu, Chen Wang, Eun Kyung Lee, Olivier Tardieu
A GPU-level analysis showing large-batch LLM inference can remain DRAM-bandwidth bound even when conventional explanations call it compute-bound.
GPU Hunter takeaway: Batch-size scaling should be benchmarked against memory bandwidth behavior, not inferred from TFLOPS alone.
Xiangbo Qi, Chaoyi Jiang, Murali Annavaram
A shared-memory kernel optimization for accelerating NF4 dequantization on NVIDIA GPUs during quantized LLM inference.
GPU Hunter takeaway: Quantized model size is not enough; GPU Hunter benchmarks should care whether the backend has fast unpacking and dequant kernels.
Shanli Xing, Yiyan Zhai, Alexander Jiang, Yixin Dong, Yong Wu, Zihao Ye
A benchmark framework connecting GPU kernel definitions, workloads, implementations, and evaluations for real inference systems.
GPU Hunter takeaway: Kernel maturity is a hardware feature in practice; the same GPU can behave very differently across inference backends.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Christopher Re
An IO-aware exact attention algorithm that reduces HBM traffic by tiling attention through SRAM.
GPU Hunter takeaway: Long context needs memory-efficient kernels; VRAM capacity alone does not guarantee usable speed.
Tri Dao
An improved FlashAttention implementation with better work partitioning, occupancy, and warp-level scheduling.
GPU Hunter takeaway: Two GPUs with similar specs can feel different when the runtime exposes better attention kernels.
Ke Hong, Guohao Dai, Jiaming Xu, Yu Wang
A decoding-focused inference engine using asynchronous softmax, flat GEMM optimization, and hardware-adaptive dataflow.
GPU Hunter takeaway: Good background for why tok/s rankings depend on decode kernels, batch size, and backend maturity.
Elias Frantar, Roberto L. Castro, Jiale Chen, Dan Alistarh
A mixed-precision kernel design for keeping 4-bit weight inference fast across useful batch sizes.
GPU Hunter takeaway: A quantized model is only as fast as the kernels that can consume its packed weights efficiently.
Ganesh Bikshandi, Jay Shah
A detailed look at implementing FlashAttention-2 on Hopper with CUTLASS, TMA, WGMMA, and fused CUDA kernels.
GPU Hunter takeaway: Architecture-specific kernel support is a real buying consideration for workstation and datacenter GPUs.
Many LLM inference paths move more data than they compute. Bandwidth, cache behavior, and kernels can dominate tokens per second.
Yes. CUDA, ROCm, Metal, and engine-specific kernels can change the effective value of a GPU for local inference.