Tag: #moe-inference6 results
FOMOE enables running the 397B Qwen3.5 MoE model at 5-9 tokens/s on a $2,100 desktop with two $500 GPUs and 32GB RAM using Q4_K_M quants. It uses VRAM caching for common experts, rolling caches, and Cache-Aware Routing (CAR) to slash NVMe reads to 7%. Achieves this with dual-GPU ping-pong and only 3.5% perplexity drop.
NVIDIA Blackwell Ultra achieves 50x tokens per MW throughput vs Hopper in DeepSeek-R1 tests, reducing million-token costs to 1/35th. It previews Rubin platform with 10x further gains. Key enablers include 130 TB/s NVLink and NVFP4 precision.
Verified benchmarks hit 198 tok/s for Qwen3.5-122B NVFP4 on 2x RTX PRO 6000 Blackwell using SGLang. Public data includes context scaling to 150K tokens without decode slowdown. Key: PCIe switch for low-latency MoE.
Benchmark shows dual AMD RX 7900 XTX GPUs achieving 123 tok/s token generation and up to 3,829 tok/s prompt processing on Qwen3.5-35B-A3B Q4_K_M using llama.cpp Vulkan backend. It outperforms single 7900 XTX and rivals high-end NVIDIA cards. vLLM on ROCm underperforms significantly with bugs.
A new PR in vLLM introduces dynamic expert caching with LRU policy, enabling 16G MoE models on 8G VRAM by keeping active experts in VRAM and offloading others to RAM. On cache misses, computation shifts to CPU while reshuffling experts to minimize latency. Upcoming features include mxfp4 quantization and disk streaming.
An infrastructure engineer shares a six- to eight-month operational review of a workstation-class AI server built for large MoE inference, creative workloads, and business research. The system combines eight RTX 3090 GPUs, two RTX 5090 GPUs, 512GB ECC RAM, and llama.cpp or koboldcpp behind Open WebUI.