Qwen3.5-35B GGUF Quants Benchmarked

🔑 Enhanced Key Takeaways

• •Unsloth conducted over 150 KL Divergence benchmarks totaling 9TB of GGUF files, identifying optimal quantization for Qwen3.5 MoE tensors like ffn_up_exps and ffn_gate_exps at 3-bit while avoiding heavy quantization on attn_* layers[1].
• •A March 5th, 2026 update to Unsloth's quantization method significantly reduced Maximum KLD for Qwen3.5-35B quants, with UD-Q4_K_XL dropping 51% from 5.894 to 2.877[1].
• •Qwen3.5-35B-A3B is a mixture-of-experts model with 35B total parameters, activating top-9 out of 256 experts per token via routing, enabling efficient compute and strong benchmarks like 84.2 on GPQA Diamond[2].

🛠️ Technical Deep Dive

• •Qwen3.5-35B-A3B uses a gated delta network for efficient long-context reasoning; GGUF Q8_1 quant (37GB) fits full GPU offload with 32K context in under 37GB VRAM, with ~0.1 perplexity loss vs BF16[2].
• •Unsloth Dynamic quants avoid MXFP4 on sensitive tensors like attn_qkv, attn_gate, ssm_beta/alpha, preferring Q4_K (4.5 bits/weight) over MXFP4 (4.25 bits/weight) for better KLD despite similar bitwidths[1][3].
• •KLD and perplexity are non-monotonic across bitwidths; e.g., Q3_K can outperform Q4_K in some cases due to tensor-specific sensitivities[1][3].
• •Optimized inference achieves 125 t/s on 16GB NVIDIA GPUs with --parallel 1 flag (10x speedup) and supports up to 155904 token contexts; 200k context at 62.98 t/s on RTX 5080 with Q4 quant[3][4].

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline