🦙Reddit r/LocalLLaMA•Mar 23, 2026Stalecollected in 2h

397B Qwen3.5 at 9 tok/s on $2100 desktop

Post LinkedIn

🦙Read original on Reddit r/LocalLLaMA

#moe-inference #local-llm #quantization #consumer-hardwarefomoeqwen3.5 fomoe

💡Run 400B MoE at 9 tok/s on $2K PC – local AI inference breakthrough

⚡ 30-Second TL;DR

What Changed

Runs 397B Qwen3.5 at 5-9 tok/s on two $500 GPUs + 32GB RAM

Why It Matters

Revolutionizes local inference for massive MoE models, making flagship AI accessible on consumer desktops without cloud dependency. Lowers barriers for builders experimenting with 400B-scale models at home.

What To Do Next

Download Qwen3.5 Q4_K_M quants and test FOMOE on dual RTX 4090 setup.

Who should care:Developers & AI Engineers

Key Points

•Runs 397B Qwen3.5 at 5-9 tok/s on two $500 GPUs + 32GB RAM
•60% VRAM hit rate reduces NVMe reads to 28%, CAR drops to 7%
•Dual GPU ping-pong overlaps loading and compute
•3.5% perplexity drop on wikitext with Cache-Aware Routing
•15K lines of C/HIP code for consumer hardware inference

🦙Read original article on Reddit r/LocalLLaMA

📰

Weekly AI Recap

Read this week's curated digest of top AI events →

👉Related Updates

Same topic

Explore #moe-inference

Same product