20 tok/s Qwen3.5-397B on M5 Max

🔑 Enhanced Key Takeaways

• •The M5 Max chip architecture utilizes a unified memory fabric that allows the 128GB RAM pool to be dynamically partitioned between the GPU and CPU, a critical factor in enabling the streaming of a 209GB model from SSD without catastrophic latency.
• •The 'flash-moe' optimization framework leverages the sparsity of Qwen3.5-397B by only loading active experts into the high-speed cache, effectively reducing the active parameter count per token to approximately 45B, which fits within the M5 Max's high-bandwidth cache.
• •The implementation of 'Temporal Expert Prediction' utilizes a lightweight heuristic model that analyzes the previous 5 tokens to predict which MoE experts will be triggered next, allowing the system to pre-fetch weights from the SSD into the cache before the compute kernel requires them.

🛠️ Technical Deep Dive

• •Model Architecture: Qwen3.5-397B is a Mixture-of-Experts (MoE) model with 397 billion total parameters and approximately 28 billion active parameters per token.
• •Quantization Strategy: Utilizes Unsloth's IQ3_XXS/IQ4_XS GGUF format, which employs importance-aware quantization to maintain perplexity while reducing the memory footprint of the expert layers.
• •IO Optimization: Employs a custom cache-io-split=4 configuration, which segments the SSD read requests into four concurrent streams to saturate the M5 Max's PCIe 5.0 bus bandwidth.
• •Kernel Fusion: The CMD2 pre-encode kernel reduces the overhead of the KV-cache management by fusing the attention calculation with the positional embedding application, saving approximately 0.44 tok/s.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline