Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inference

A peer-to-peer GPU cache management framework that uses high-bandwidth GPU interconnects to reduce host-memory offload latency. Multi-GPU inference is often limited by where model state and KV tensors live, not just aggregate VRAM on paper.

This paper helps explain why a dual-GPU workstation is not automatically twice as useful as one stronger card.

Multi-GPU inference is often limited by where model state and KV tensors live, not just aggregate VRAM on paper. The useful part for GPU Hunter readers is not the abstract result alone; it is the hardware implication: whether a model fits, whether a runtime can use the format, or whether throughput is limited by memory movement instead of arithmetic.

For dual-GPU and workstation setups, interconnect bandwidth and cache placement can change whether extra GPUs actually help. For shared inference, the important question is not only how fast one prompt runs. Batching, scheduling, cache placement, and request mix decide whether a GPU behaves like a reliable service.