Qwen3.5-35B Tops VLLM A100 Benchmarks

🔑 Enhanced Key Takeaways

• •Qwen3.5-35B-A3B employs a Gated DeltaNet hybrid architecture with native multimodal support and extended context up to 1M tokens, enhancing its deployment flexibility beyond pure text inference[2][7].
• •The model achieves top benchmark scores including SWE-bench 72.4 (matching GPT-5-mini), LiveCodeBench 80.7, and IFEval 95.0, establishing it as the leading medium-sized model for coding and instruction-following[2].
• •At BF16 precision, Qwen3.5-35B-A3B requires approximately 54GB VRAM on a single A100 80GB GPU, dropping to 14GB with 4-bit AWQ quantization for consumer GPUs like RTX 4090[2].
• •Qwen3.5 series introduces native FP8 pipelines reducing memory by 50% and boosting speeds over 10% at trillion-token scale, contributing to its vLLM efficiency[4].

🛠️ Technical Deep Dive

• •Qwen3.5-35B-A3B is a Mixture-of-Experts (MoE) model with 3B active parameters out of 35B total, using Gated DeltaNet hybrid architecture for efficient routing and native multimodal (text+video) fusion[2][7].
• •FlashInfer attention in the benchmark replaces FlashAttn, yielding higher throughput (352 tok/s output) due to optimized kernel for AWQ-4bit quantization and lower KV cache overhead on dual A100 40GB with tensor-parallel=2[1].
• •Quantization variants include AWQ-4bit (supports max concurrency 45) and FP8; FP8 reduces memory by 50% via native pipeline while maintaining >10% speed gains at scale[4].
• •64 layers enable deep reasoning (e.g., HMMT 92.0, DynaMath 87.7); 7 quantized HuggingFace variants available for consumer deployment[2].
• •Prefill speed advantage stems from reduced KV cache usage in MoE design, allowing higher total tok/s (1357) compared to dense Qwen3-30B variants[2].

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline