35B A3B MoE Faster Than 9B Locally

Post LinkedIn

🦙Read original on Reddit r/LocalLLaMA

#local-inference#vram#moe#tokens-per-secondqwen-3.5-a3b-moeqwen-3.5a3b-moellama.cpp

💡Real-world speeds: 35B MoE at 55t/s on 16GB VRAM beats slower 9B

⚡ 30-Second TL;DR

What Changed

35B A3B MoE runs at steady 49-55 tokens/s on 16GB VRAM and 64GB RAM

Why It Matters

Highlights practical local run speeds for MoE models on consumer hardware, guiding hardware choices for LLM inference.

What To Do Next

Test Qwen 3.5 35B-A3B MoE on your 16GB VRAM setup using llama.cpp.

Who should care:Developers & AI Engineers

Key Points

•35B A3B MoE runs at steady 49-55 tokens/s on 16GB VRAM and 64GB RAM
•9B variant slower at 23 t/s on same hardware
•User praises 35B MoE as amazing for local inference
•9B rumored better than 120B OSS despite speed

🧠 Deep Insight

Background and context from public sources — not the original article. 6 sources cited.

🔑 Enhanced Key Takeaways

•Qwen3.5-35B-A3B is a Mixture-of-Experts (MoE) model with 35B total parameters but only activates ~3B parameters per token via a routing system, enabling high speed on consumer hardware.[1]
•Qwen3.5 series, including the 35B-A3B MoE, was officially released by Alibaba's Qwen team on February 24, 2026, and made available on Hugging Face.[5][6]
•The 35B-A3B excels in speed (60-100+ t/s on RTX 3090/4090) but trails dense models like Qwen3.5-27B (15-25 t/s) in logic, coding, and complex reasoning due to fewer active parameters.[1]

🛠️ Technical Deep Dive

•Qwen3.5-35B-A3B uses MoE architecture where a router activates ~3B parameters out of 35B total for each token, approximating the compute of a 3B dense model while retaining larger model knowledge.[1]
•Fine-tuning Qwen3.5-35B-A3B requires 74GB VRAM for bf16 LoRA; Unsloth enables 1.5x faster training with 50% less VRAM, supports MoE kernels, and recommends bf16 over 4-bit QLoRA for stability.[3]
•Community estimates effective intelligence via √(Total × Active) formula, placing 35B-A3B ~10B dense equivalent, between 7B-14B in capability.[1]

🔮 Future ImplicationsAI analysis grounded in cited sources

MoE adoption will grow for local inference on 16-24GB VRAM setups

Models like Qwen3.5-35B-A3B demonstrate 2-4x speed gains over dense equivalents while fitting consumer hardware, as validated by user benchmarks and fine-tuning optimizations.[1][3]

Qwen3.5 MoE variants will dominate lightweight agentic tasks

Similar A3B MoE models like GLM-4.7-Flash already excel in benchmarks for tool use and long-context reasoning, suggesting Qwen3.5-35B-A3B follows this efficiency trend.[2]