Parallel CPU-GPU Execution for LLM Inference on Constrained GPUs

A hybrid CPU-GPU execution approach that overlaps CPU-offloaded KV and attention work with GPU execution during memory-bound decoding. Constrained GPUs are the norm for local users, and naive offload often loses to PCIe and scheduling overhead.

This paper gives practical caution for budget GPU buyers who assume offload will make every large model usable.

Constrained GPUs are the norm for local users, and naive offload often loses to PCIe and scheduling overhead. The useful part for GPU Hunter readers is not the abstract result alone; it is the hardware implication: whether a model fits, whether a runtime can use the format, or whether throughput is limited by memory movement instead of arithmetic.

Offload is only useful when CPU work, GPU kernels, and transfers overlap; otherwise it just makes a barely fitting model slow. For one-box local AI, the practical issue is how model format, runtime, memory hierarchy, and offload policy interact. This is where a cheaper GPU can be a good choice or a frustrating compromise.