Why Is Muon Only for Transformers?

🔑 Enhanced Key Takeaways

• •Muon (Momentum-Unnormalized Optimizer) relies on a second-order approximation that performs a singular value decomposition (SVD) on weight updates, which is computationally expensive and memory-intensive for non-Transformer architectures.
• •The optimizer's effectiveness is tightly coupled with the specific weight-tying and parameter-sharing properties of Transformer layers, which do not translate directly to the hierarchical, spatial-feature-focused structure of Convolutional Neural Networks.
• •Recent research suggests that Muon's performance gains are highly sensitive to the specific learning rate schedules and batch sizes used in LLM pre-training, making it difficult to tune for the diverse training dynamics of vision models.

🛠️ Technical Deep Dive

• •Muon performs a momentum-based update followed by a projection step that enforces orthogonality on the weight update matrix.
• •The core operation involves computing the SVD of the momentum matrix M, specifically M = UΣVᵀ, and replacing it with U Vᵀ to ensure the update is a rotation/reflection.
• •It requires significantly more memory than AdamW due to the need to store and compute SVDs for large weight matrices, limiting its application to architectures where weight matrices are large enough to justify the overhead.
• •The optimizer is typically applied only to the dense layers of Transformers, as the SVD operation is not well-defined or efficient for the smaller, heterogeneous kernels found in ConvNets.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline