Qwen3.6 35B-A3B Runs Fast on 780M iGPU

🔑 Enhanced Key Takeaways

• •The Qwen3.6 series utilizes a novel 'Active-3-Base' (A3B) Mixture-of-Experts architecture, which dynamically routes tokens through only 3 billion parameters per forward pass, significantly reducing the VRAM footprint compared to dense 35B models.
• •The performance gains on the Radeon 780M are largely attributed to recent upstream patches in llama.cpp that optimize Vulkan memory management for unified memory architectures, specifically addressing the overhead of GTT (Graphics Translation Table) mapping.
• •Community testing indicates that while the 780M iGPU can handle the Q6_K quantization, the bottleneck shifts to system memory bandwidth (DDR5-5600/6400), making high-speed RAM essential for maintaining the reported 20 tg/s generation speed.

🛠️ Technical Deep Dive

• •Model Architecture: Qwen3.6 35B-A3B uses a sparse MoE design where the total parameter count is 35B, but only 3B parameters are active per token, allowing it to fit into system RAM while providing the reasoning capabilities of a much larger model.
• •Vulkan Implementation: The llama.cpp Vulkan backend utilizes the 'VK_KHR_buffer_device_address' extension to minimize CPU-GPU synchronization overhead.
• •Kernel Tweaks: The 'GTT' (Graphics Translation Table) adjustment involves increasing the i915.enable_gtt or equivalent AMD amdgpu.gttsize parameter in the Linux kernel boot arguments to allow the iGPU to map larger portions of system RAM as VRAM.
• •Quantization: The Q6_K (6-bit) quantization format provides a balance between perplexity retention and memory bandwidth utilization, which is critical for iGPU performance where memory bus width is the primary constraint.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline