Curated local AI inference papers covering llama.cpp quantization, constrained GPUs, Apple Silicon, offload, and single-workstation LLM systems.
Local inference is not just cloud serving on smaller hardware. Desktop GPUs, used cards, laptops, and Apple Silicon machines have different memory hierarchies, runtimes, quantization formats, and offload limits.
This cluster focuses on the papers that help GPU Hunter answer the practical question: what can you run on one machine, and what compromises are required when the model barely fits?
Read these first if you want the fastest path from research to a hardware decision.
Uygar Kurt
A unified empirical evaluation of llama.cpp quantization formats for Llama-3.1-8B-Instruct on commodity hardware.
GPU Hunter takeaway: A GPU recommendation is incomplete without naming the quantization formats that are realistic for that VRAM tier.
Afsara Benazir, Felix Xiaozhu Lin
A profiling study of Apple Silicon's unified memory architecture for on-device LLM inference under different quantization choices.
GPU Hunter takeaway: Mac recommendations should compare quantized throughput, memory pressure, and unified-memory behavior against CUDA GPUs.
Yixin Song, Zeyu Mi, Haotong Xie, Haibo Chen
A consumer-GPU inference engine that exploits activation locality and CPU-GPU hybrid execution.
GPU Hunter takeaway: A 4090-class card can punch above its VRAM limit when the runtime is designed around model sparsity and locality.
These papers go deeper into kernels, cache policy, low-bit formats, or serving tradeoffs.
He Sun, Li Li, Mingjun Xiao, Chengzhong Xu
LeoAM uses adaptive hierarchical GPU-CPU-disk KV management, lightweight KV abstracts, compression, and pipelining for long context on one commodity GPU.
GPU Hunter takeaway: A single desktop GPU can handle longer contexts when the runtime manages GPU, CPU, and disk tiers deliberately.
Jiakun Fan, Yanglin Zhang, Xiangchen Li, Dimitrios S. Nikolopoulos
A hybrid CPU-GPU execution approach that overlaps CPU-offloaded KV and attention work with GPU execution during memory-bound decoding.
GPU Hunter takeaway: Offload is only useful when CPU work, GPU kernels, and transfers overlap; otherwise it just makes a barely fitting model slow.
Ying Sheng, Lianmin Zheng, Binhang Yuan, Ion Stoica
A system for running very large models on limited hardware by coordinating GPU, CPU, and disk memory.
GPU Hunter takeaway: Offloading can make a model fit, but it should be treated as a throughput compromise, not free VRAM.
The complete paper set for this topic, with source links and GPU Hunter takeaways.
Aman Sunesh, Ali Alshehhi, Hivansh Dhakne
A single-GPU controller that routes each request across FP16, quantized, speculative, prefix-cached, and batched inference modes using cheap workload features.
GPU Hunter takeaway: A desktop GPU can serve different prompts better when the runtime switches strategy per request instead of locking into one mode.
Uygar Kurt
A unified empirical evaluation of llama.cpp quantization formats for Llama-3.1-8B-Instruct on commodity hardware.
GPU Hunter takeaway: A GPU recommendation is incomplete without naming the quantization formats that are realistic for that VRAM tier.
Afsara Benazir, Felix Xiaozhu Lin
A profiling study of Apple Silicon's unified memory architecture for on-device LLM inference under different quantization choices.
GPU Hunter takeaway: Mac recommendations should compare quantized throughput, memory pressure, and unified-memory behavior against CUDA GPUs.
He Sun, Li Li, Mingjun Xiao, Chengzhong Xu
LeoAM uses adaptive hierarchical GPU-CPU-disk KV management, lightweight KV abstracts, compression, and pipelining for long context on one commodity GPU.
GPU Hunter takeaway: A single desktop GPU can handle longer contexts when the runtime manages GPU, CPU, and disk tiers deliberately.
Jiakun Fan, Yanglin Zhang, Xiangchen Li, Dimitrios S. Nikolopoulos
A hybrid CPU-GPU execution approach that overlaps CPU-offloaded KV and attention work with GPU execution during memory-bound decoding.
GPU Hunter takeaway: Offload is only useful when CPU work, GPU kernels, and transfers overlap; otherwise it just makes a barely fitting model slow.
Yixin Song, Zeyu Mi, Haotong Xie, Haibo Chen
A consumer-GPU inference engine that exploits activation locality and CPU-GPU hybrid execution.
GPU Hunter takeaway: A 4090-class card can punch above its VRAM limit when the runtime is designed around model sparsity and locality.
Ying Sheng, Lianmin Zheng, Binhang Yuan, Ion Stoica
A system for running very large models on limited hardware by coordinating GPU, CPU, and disk memory.
GPU Hunter takeaway: Offloading can make a model fit, but it should be treated as a throughput compromise, not free VRAM.
Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Mehrdad Farajtabar
A hardware-aware method for running models larger than available DRAM by optimizing flash-memory transfers.
GPU Hunter takeaway: Model fit is not binary; memory hierarchy determines whether a barely fitting setup is usable or painful.
Start with the llama.cpp quantization evaluation, PowerInfer, and the constrained-GPU execution papers. They map directly to VRAM fit, offload, and runtime expectations.
Only partly. Apple Silicon uses unified memory and Metal/MLX-style runtimes, so capacity comparisons need separate throughput and backend context.