ModeSwitch-LLM: A Lightweight Phase-Aware Controller for Cross-Mode LLM Inference on a Single GPU

A single-GPU controller that routes each request across FP16, quantized, speculative, prefix-cached, and batched inference modes using cheap workload features. It treats local inference as a dynamic operating problem instead of a one-time model loading choice.

ModeSwitch-LLM is a good bridge from paper research to GPU Hunter's buying workflow because it explains why one benchmark number cannot describe every local inference workload.

It treats local inference as a dynamic operating problem instead of a one-time model loading choice. The useful part for GPU Hunter readers is not the abstract result alone; it is the hardware implication: whether a model fits, whether a runtime can use the format, or whether throughput is limited by memory movement instead of arithmetic.

A desktop GPU can serve different prompts better when the runtime switches strategy per request instead of locking into one mode. For one-box local AI, the practical issue is how model format, runtime, memory hierarchy, and offload policy interact. This is where a cheaper GPU can be a good choice or a frustrating compromise.