Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference

A GPU-level analysis showing large-batch LLM inference can remain DRAM-bandwidth bound even when conventional explanations call it compute-bound. It reinforces GPU Hunter's central point: memory bandwidth and data movement often decide inference performance.

This paper is foundational for GPU Hunter's emphasis on memory bandwidth and not just tensor-core peak numbers.

It reinforces GPU Hunter's central point: memory bandwidth and data movement often decide inference performance. The useful part for GPU Hunter readers is not the abstract result alone; it is the hardware implication: whether a model fits, whether a runtime can use the format, or whether throughput is limited by memory movement instead of arithmetic.

Batch-size scaling should be benchmarked against memory bandwidth behavior, not inferred from TFLOPS alone. For GPU buyers, this points at the gap between spec-sheet compute and real inference speed. Kernels, memory traffic, and backend support determine how much of the hardware is actually usable.