Multi-GPU KV Cache and Expert Offload Query

🔑 Enhanced Key Takeaways

• •llama.cpp supports automatic layer offloading with manual control via num_gpu parameter, allowing users to specify exactly how many transformer layers load to GPU versus CPU, enabling fine-grained KV cache distribution across heterogeneous GPU setups[2]
• •Mixture-of-Experts (MoE) models like the 35B-A3B variant require only ~200MB of KV cache per concurrent user at 8K context due to Grouped Query Attention (GQA) combined with sparse expert activation, making them viable for weak+strong GPU pairs where experts can be strategically offloaded[3]
• •Multi-GPU scaling in llama.cpp without NVLink yields approximately 1.6x throughput from 2 GPUs due to PCIe bandwidth bottlenecks; pipeline parallelism (PP) is recommended over tensor parallelism (TP) for heterogeneous setups to avoid layer-splitting inefficiencies[1]
• •CUDA_VISIBLE_DEVICES environment variable enables per-GPU assignment in Ollama and llama.cpp, allowing model layers to be distributed across multiple GPUs with different VRAM capacities by controlling which GPU processes which layers[2]

🛠️ Technical Deep Dive

• •KV Cache Management: Each concurrent inference request in llama.cpp carves out an independent KV cache memory slot; MoE models with GQA achieve ~200MB per user at 8K context, enabling multi-user concurrency on 24GB GPUs with 4+ simultaneous requests while maintaining ~18 tokens/sec per user[3]
• •Layer Offloading Mechanics: Ollama automatically determines layer distribution based on available VRAM; manual num_gpu N parameter forces exactly N layers to GPU with remainder on CPU, reducing VRAM footprint (e.g., 4.8GB vs 7.2GB on RTX 4060 8GB) but incurring performance penalties[2]
• •Expert Tensor Distribution: MoE models activate only ~3B parameters per token despite larger total parameter count; this sparse activation pattern allows expert tensors to be offloaded to weaker GPUs while non-expert layers remain on stronger GPU without proportional performance degradation[3]
• •Multi-GPU Topology Optimization: PCIe configuration requires x8 lanes minimum per GPU (x4 creates bottlenecks); NUMA topology awareness via nvidia-smi topo -m is critical for latency reduction; NVLink (RTX 3090) provides superior inter-GPU bandwidth compared to PCIe-only setups[1]
• •Quantization Impact on Offloading: Q4_K_M quantization reduces VRAM by 4x with minimal quality loss; FP8 (supported on H100/H200/B200) halves memory usage, enabling larger models to fit entirely on GPU and reducing reliance on CPU offloading[4]

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline