Qwen 3.5 397B Beats Trillion-Param Rival Cheaply

• Architecture: 60 layers with hidden dimension 4,096; layout alternates 15 blocks of (3× Gated DeltaNet→MoE) and (1× Gated Attention→MoE)[1] • Attention Mechanism: Gated DeltaNet uses 64 linear attention heads for values, 16 for QK with 128-dim heads; Gated Attention uses 32 heads for Q, 2 for KV with 256-dim heads and RoPE dimension 64[1] • Expert Configuration: 512 total experts with 1,024 intermediate dimension; 11 experts activated per token (10 routed + 1 shared)[1] • Context Extension: Native 262,144 token input context length, extensible to 1,010,000 tokens via YaRN RoPE scaling[1] • Vocabulary: 248,320 tokens[1] • Memory Requirements: ~800GB VRAM for full FP16/BF16 model; ~220GB for 4-bit quantization; runnable on Mac Studio/Pro with M-series Ultra (256GB RAM) or multi-GPU clusters[2] • Training Data: Trillions of multimodal tokens across image, text, and video; 201 languages and dialects; training labeling and collection automated[1] • Inference Modes: Thinking mode (internal reasoning) and Fast mode for standard workflows[2]

Qwen3.5-397B-A17B represents a significant shift in the open-source LLM landscape by demonstrating that sparse mixture-of-experts architectures can match or exceed dense trillion-parameter models in reasoning and coding while dramatically reducing computational costs and inference latency. This challenges the prevailing assumption that scale alone determines capability, potentially accelerating adoption of efficient open-weight models in enterprise and research settings. The native multimodal training from inception—rather than bolted-on vision adapters—establishes a new standard for vision-language model design. The 1M token context window in the hosted Qwen3.5-Plus variant enables new use cases in document analysis, long-form reasoning, and agentic workflows. However, the MMMLU gap versus Gemini 3 Pro (88.5 vs 90.6) suggests frontier performance still favors proprietary models, though the cost and speed advantages may offset this for many applications. The model's availability as open-weight software could intensify competition in the API market and influence how other labs (DeepSeek, Minimax, Kimi) design their next-generation models.