Qwen 35B-A3B: 26 t/s on 8GB Laptop at 100K Context

🔑 Enhanced Key Takeaways

• •Qwen3.5-35B-A3B is a native vision-language model supporting text and image inputs with a native context length of 262,144 tokens, extensible to over 1 million tokens[1][3][4].
• •Model architecture features a hybrid design with 40 layers, hidden dimension of 2048, 256 Mixture-of-Experts (8 routed + 1 shared activated), and Gated Delta Networks for efficient inference[1].
• •On high-end GPUs like RTX 3090/4090, users report 60-100+ t/s generation speeds due to only ~3B active parameters per token in its MoE setup[2].
• •Released on February 24, 2026, it achieves strong benchmarks like 84.5% on GPQA and 37.1 Intelligence Index score[4].

🛠️ Technical Deep Dive

• •Total parameters: 35B, active parameters: ~3B; hidden dimension: 2048; token embedding: 248,320 (padded)[1][2].
• •Architecture: 40 layers in layout 10 × (3 × (Gated DeltaNet → MoE) → 1 × (Gated Attention → MoE)); Gated DeltaNet uses 32 linear attention heads for V and 16 for QK[1].
• •Gated Attention: 16 heads for Q, 2 for KV; head dimension 256; Rotary Position Embedding dimension 64; Mixture-of-Experts: 256 experts, 8 routed + 1 shared activated; expert intermediate dimension 512[1].
• •Multimodal capabilities: Unified vision-language foundation via early fusion training on multimodal tokens; supports text/image/video input, text output[1][4].
• •Context: 262,144 tokens natively, extensible to 1,010,000; trained with multi-step MTP[1][3][4].

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline