MiniMax M2.7 GGUF NaN Fixes and Benchmarks

🔑 Enhanced Key Takeaways

• •The NaN issue stems from the MiniMax-M2.7 model's use of a specific Mixture-of-Experts (MoE) architecture that utilizes high-precision activations, which overflow when quantized using standard llama.cpp K-quants.
• •The CUDA 13.2 incompatibility is linked to a regression in the cuBLAS library's handling of sub-8-bit integer matrix multiplication kernels, specifically affecting models with non-standard expert routing.
• •Community efforts have identified that disabling 'expert-parallel' optimizations in llama.cpp during the quantization process mitigates the overflow risk, even without full re-quantization.

🛠️ Technical Deep Dive

• •Model Architecture: MiniMax-M2.7 utilizes a sparse MoE structure with 2.7 billion active parameters, featuring a unique 'expert-down-projection' layer that is highly sensitive to quantization noise.
• •Overflow Mechanism: The 'blk.61.ffn_down_exps' layer exhibits extreme activation values during inference, exceeding the dynamic range of Q4_K_S and Q5_K_M quantization schemes.
• •Quantization Mitigation: The fix involves forcing the specific problematic layers to remain in FP16 or using a higher-precision I-quant (e.g., IQ4_XS) to preserve the activation distribution.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline