# GPU Hunter > The GPU index for local AI inference ## What is GPU Hunter? GPU Hunter is a benchmark aggregation index for GPUs used in local AI inference. We source and normalize published community benchmark data for consumer GPUs from NVIDIA, AMD (ROCm), and Intel (Arc), workstation cards (Blackwell, Ada, Ampere), Apple Silicon, and NVIDIA DGX Spark to help engineers find hardware that clears their model-memory plan. ## Key Data Points - GPUs tracked: 20 - Models profiled: 10 - Benchmark data sourced from llama.cpp community benchmarks (GitHub discussions #15013, #4167, #15021, #16578), Hardware Corner, and AwesomeAgents leaderboard ## GPU Index - RTX PRO 6000 Blackwell: 96GB VRAM, 1792GB/s bandwidth, 141 tok/s (Llama 8B Q4), $8499 dated reference estimate - GeForce RTX 5090: 32GB VRAM, 1792GB/s bandwidth, 145 tok/s (Llama 8B Q4), $1999 dated reference estimate - GeForce RTX 4090: 24GB VRAM, 1008GB/s bandwidth, 104 tok/s (Llama 8B Q4), $1799 dated reference estimate - GeForce RTX 3090: 24GB VRAM, 936GB/s bandwidth, 87 tok/s (Llama 8B Q4), $749 dated reference estimate - NVIDIA DGX Spark: 128GB VRAM, 273GB/s bandwidth, 45 tok/s (Llama 8B Q4), $3999 dated reference estimate - Apple M3 Ultra: 512GB VRAM, 819GB/s bandwidth, 92 tok/s (Llama 8B Q4), $9499 dated reference estimate - Apple M4 Max: 128GB VRAM, 546GB/s bandwidth, 83 tok/s (Llama 8B Q4), $4699 dated reference estimate - GeForce RTX 5080: 16GB VRAM, 960GB/s bandwidth, 92 tok/s (Llama 8B Q4), $999 dated reference estimate - GeForce RTX 5070 Ti: 16GB VRAM, 896GB/s bandwidth, 86 tok/s (Llama 8B Q4), $749 dated reference estimate - GeForce RTX 5070: 12GB VRAM, 672GB/s bandwidth, 65 tok/s (Llama 8B Q4), $549 dated reference estimate - GeForce RTX 4080 SUPER: 16GB VRAM, 736GB/s bandwidth, 78 tok/s (Llama 8B Q4), $899 dated reference estimate - GeForce RTX 4070 Ti SUPER: 16GB VRAM, 672GB/s bandwidth, 70 tok/s (Llama 8B Q4), $699 dated reference estimate - GeForce RTX 3090 Ti: 24GB VRAM, 1008GB/s bandwidth, 94 tok/s (Llama 8B Q4), $849 dated reference estimate - GeForce RTX 3060 12GB: 12GB VRAM, 360GB/s bandwidth, 40 tok/s (Llama 8B Q4), $249 dated reference estimate - Radeon RX 7900 XTX: 24GB VRAM, 960GB/s bandwidth, 66 tok/s (Llama 8B Q4), $849 dated reference estimate - Radeon RX 9070 XT: 16GB VRAM, 512GB/s bandwidth, 56 tok/s (Llama 8B Q4), $549 dated reference estimate - Intel Arc B580: 12GB VRAM, 456GB/s bandwidth, 35 tok/s (Llama 8B Q4), $249 dated reference estimate - Apple M4 Pro: 48GB VRAM, 273GB/s bandwidth, 51 tok/s (Llama 8B Q4), $2499 dated reference estimate - NVIDIA RTX A6000: 48GB VRAM, 768GB/s bandwidth, 73 tok/s (Llama 8B Q4), $2499 dated reference estimate - NVIDIA RTX 6000 Ada: 48GB VRAM, 960GB/s bandwidth, 95 tok/s (Llama 8B Q4), $6800 dated reference estimate ## Model VRAM Requirements - Qwen3 32B: weight estimates Q4=19GB, Q8=36GB, FP16=64GB; runtime and KV cache need additional VRAM - Qwen2.5 72B: weight estimates Q4=44GB, Q8=77.5GB, FP16=145GB; runtime and KV cache need additional VRAM - Qwen3 235B-A22B: weight estimates Q4=132GB, Q8=240GB, FP16=470GB; runtime and KV cache need additional VRAM - Llama 3.3 70B: weight estimates Q4=40GB, Q8=75GB, FP16=140GB; runtime and KV cache need additional VRAM - DeepSeek V3: weight estimates Q4=380GB, Q8=700GB, FP16=1300GB; runtime and KV cache need additional VRAM - Llama 3.1 8B: weight estimates Q4=5GB, Q8=9GB, FP16=16GB; runtime and KV cache need additional VRAM - Qwen3 14B: weight estimates Q4=8GB, Q8=15GB, FP16=28GB; runtime and KV cache need additional VRAM - Mistral 7B: weight estimates Q4=4GB, Q8=8GB, FP16=14GB; runtime and KV cache need additional VRAM - Gemma 2 27B: weight estimates Q4=16GB, Q8=30GB, FP16=54GB; runtime and KV cache need additional VRAM - Codestral 22B: weight estimates Q4=13GB, Q8=24GB, FP16=44GB; runtime and KV cache need additional VRAM ## Pages - /browse — Full GPU index with filters, sorting, and budget calculator - /compare — Side-by-side GPU comparison tool - /tools — Local AI planning tools hub - /tools/llm-vram-calculator — Model-first VRAM planning calculator with runtime headroom - /tools/gpu-cost-calculator — GPU electricity, ownership cost, effective hourly cost, and manual buy-versus-cloud scenario calculator - /alerts — Reference-only workload and GPU watch-plan builder; no live monitoring or notifications - /gpu/[id] — Detailed GPU benchmark pages with specs, model fit, and pricing - /model/[id] — Model VRAM requirements and compatible GPU recommendations (10 models) - /blog — Hardware benchmarks, buying guides, and technical write-ups - /research — Curated arXiv research library for local AI inference, quantization, GPU kernels, KV cache optimization, and LLM serving systems - /llms-full.txt — Expanded crawler and LLM-readable summary of GPU Hunter data, research clusters, and canonical URLs ## Research Topic Pages - /research/llm-quantization-papers — LLM quantization papers: Curated LLM quantization papers covering GPTQ, AWQ, GGUF, NF4, FP4, NVFP4, MXFP4, and what low-bit inference means for GPU choice. - /research/kv-cache-optimization-papers — KV cache optimization papers: Research papers on KV cache quantization, compression, eviction, offload, reuse, and long-context inference bottlenecks for local and served LLMs. - /research/local-ai-inference-papers — local AI inference papers: Curated local AI inference papers covering llama.cpp quantization, constrained GPUs, Apple Silicon, offload, and single-workstation LLM systems. - /research/gpu-inference-optimization-papers — GPU inference optimization papers: GPU inference optimization papers on memory bandwidth, FlashAttention, NF4 dequantization, kernel maturity, batching, and LLM bottlenecks. - /research/llm-serving-systems-papers — LLM serving systems papers: Curated LLM serving systems papers on vLLM, PagedAttention, speculative decoding, batching, cache reuse, scheduling, and GPU server design. - /research/2026-llm-inference-papers — 2026 LLM inference papers: Fresh 2026 LLM inference papers on FP4, KV cache quantization, local inference controllers, GPU kernels, AMD serving, and long-context systems. ## Research Year Pages - /research/year/2026 — 2026 LLM Inference Research Papers: Curated 2026 LLM inference research across local AI, FP4 quantization, KV cache optimization, kernels, AMD serving, and single-GPU systems. - /research/year/2025 — 2025 LLM Inference Research Papers: Curated 2025 LLM inference papers covering KV cache compression, Apple Silicon profiling, constrained GPUs, serving systems, and memory bottlenecks. ## High-Value Research Paper Pages - /research/papers/2605.23057 — ModeSwitch-LLM: A Lightweight Phase-Aware Controller for Cross-Mode LLM Inference on a Single GPU; single GPU LLM inference optimization - /research/papers/2605.17757 — OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization; 2-bit KV cache quantization - /research/papers/2604.19157 — SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving; 4-bit KV cache quantization - /research/papers/2604.05012 — Comparative Characterization of KV Cache Management Strategies for LLM Inference; KV cache management strategies - /research/papers/2604.02556 — Fast NF4 Dequantization Kernels for Large Language Model Inference; NF4 dequantization kernels - /research/papers/2603.10031 — Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study; AMD Instinct LLM inference optimization - /research/papers/2603.08747 — Diagnosing FP4 inference: a layer-wise and block-wise sensitivity analysis of NVFP4 and MXFP4; FP4 inference sensitivity analysis - /research/papers/2603.20397 — KV Cache Optimization Strategies for Scalable and Efficient LLM Inference; KV cache optimization strategies - /research/papers/2602.08005 — DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity; residual KV cache compression - /research/papers/2602.00328 — Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inference; peer-to-peer GPU caching for LLM inference ## How to Use 1. Pick a model you want to run (e.g., Qwen2.5 72B) 2. Choose your quantization (Q4, Q8, FP16) 3. GPU Hunter adds planning headroom, then ranks compatible GPUs by performance and dated reference cost 4. Model ownership economics with assumptions you control 5. Use the compare tool or save a local reference-only watch plan for your shortlist ## Contact Website: https://www.gpuhunter.io Partnerships: https://www.gpuhunter.io/partnerships Email: partnerships@gpuhunter.io