DeepSpeed Now Supports Muon Optimizer

🔑 Enhanced Key Takeaways

•The Muon optimizer achieves approximately 2x computational efficiency compared to AdamW for compute-optimal training, potentially cutting LLM training costs by around 50%.
•Muon specifically targets 2D weight matrices (e.g., linear layers in transformers) by applying Newton-Schulz orthogonalization to momentum-based updates, requiring a hybrid approach where other parameters like embeddings and biases are still optimized by traditional methods like AdamW.
•This optimizer addresses 'spectral bias' in conventional optimizers by orthogonalizing gradients, which promotes balanced learning across all feature directions and prevents over-optimization of high-variance features.
•DeepSpeed's integration of Muon includes distributed training compatibility, enabling the partitioning of optimizer state within data parallel groups and supporting flattened and padded parameters for ZeRO stages 1, 2, and 3.
•Muon has demonstrated scalability from smaller models to those with trillion parameters, with specialized variants like MuonClip developed to enhance stability during training at extreme scales.

📊 Competitor Analysis▸ Show

Feature/Metric	DeepSpeed	FSDP (PyTorch)	Muon Optimizer	AdamW Optimizer
Primary Focus	Large-scale distributed training & inference optimization	Memory-efficient data parallelism for large models	Matrix-aware optimization for 2D parameters	General-purpose adaptive learning rate optimization
Memory Optimization	ZeRO (Zero Redundancy Optimizer) partitions optimizer states, gradients, parameters; ZeRO-Offload	Fully Sharded Data Parallelism (FSDP) shards optimizer states, gradients, parameters	33% memory savings for optimizer states (2 copies vs. 3)	Requires 3 copies of parameters (model, momentum, variance)
Parallelism	3D Parallelism (Data, Pipeline, Tensor Slicing)	Data Parallelism with sharding	N/A (optimizer, not a parallelism framework)	N/A (optimizer, not a parallelism framework)
Efficiency Gains	Up to 10x faster training, 6.2x throughput improvement, 2.8x faster convergence	Excels in mid-range (100M-1B parameters)	~2x computational efficiency vs. AdamW for LLMs	Widely used, but less efficient for large LLMs
Scalability	Trillion+ parameters	Up to 200B parameters, excels in mid-range	Proven for trillion-parameter models	Can struggle with efficiency at extreme scales
Integration	PyTorch, HuggingFace	PyTorch native	DeepSpeed, PyTorch, NVIDIA NeMo, Tinygrad	PyTorch, TensorFlow, etc. (ubiquitous)
Computational Overhead	Optimized system kernels, mixed precision	Less runtime bookkeeping, fused communication	~0.5% to 1% for Newton-Schulz iterations	Generally lower per-step overhead than Muon, but higher overall FLOPs for convergence

🛠️ Technical Deep Dive

Muon's Core Mechanism: Muon (MomentUm Orthogonalized by Newton-Schulz) operates by taking the updates generated by SGD-momentum and applying a Newton-Schulz (NS) iteration as a post-processing step to each 2D parameter matrix. This process effectively replaces the update matrix with its nearest semi-orthogonal matrix, which helps in balancing learning across all feature directions by equalizing singular values.
Memory and Computational Trade-offs: Muon reduces memory consumption for optimizer states by approximately 33% compared to AdamW (requiring 2 copies of parameters instead of 3). This memory saving comes with a minor computational overhead of about 0.5% to 1% of the total training FLOPs due to the Newton-Schulz iterations.
Hybrid Optimization Requirement: Muon is specifically designed for 2D weight matrices, such as linear layers in transformer models or convolutional filters. For other types of parameters, including embeddings, biases, and output/classifier heads, it is empirically found that traditional optimizers like AdamW should still be used to achieve optimal performance.
DeepSpeed's Optimization Suite: DeepSpeed enhances training efficiency through several key innovations. Its Zero Redundancy Optimizer (ZeRO) partitions optimizer states, gradients, and parameters across data-parallel processes, significantly reducing memory footprint. DeepSpeed also employs 3D parallelism, combining data, pipeline, and tensor-slicing parallelism, to enable the training of models with billions to trillions of parameters.
Integration Specifics: The DeepSpeed integration of Muon ensures compatibility with distributed training by partitioning the optimizer state within data parallel groups and managing process groups. It also supports flattened and padded parameters, crucial for DeepSpeed's ZeRO stages 1, 2, and 3.

🔮 Future ImplicationsAI analysis grounded in cited sources

Broader adoption of matrix-aware optimizers will accelerate large language model development.

Muon's demonstrated efficiency and scalability for LLMs, coupled with its integration into major frameworks like DeepSpeed and PyTorch, indicates a growing industry shift towards optimizers that leverage the geometric structure of neural network weights for improved training performance.

AI training costs for large models will significantly decrease, democratizing access to frontier AI research.

Muon's ability to achieve comparable model performance with roughly half the computational cost of AdamW, combined with DeepSpeed's memory and compute optimizations, will make training larger and more complex models more accessible to a wider range of organizations.

Research into optimizer biases and their impact on model behavior will intensify.

While Muon offers significant speedups, recent research suggests it removes a 'simplicity bias' inherent in older optimizers like SGD, potentially leading to models that struggle to uncover common underlying structures or are more prone to fitting spurious features, prompting further investigation into these trade-offs.

⏳ Timeline

2020-02

Microsoft introduces DeepSpeed and the ZeRO (Zero Redundancy Optimizer) technology.

2020-09

DeepSpeed announces 3D parallelism for trillion-parameter models and ZeRO-Offload for enhanced single-GPU training.

2024-10

Keller Jordan introduces the Muon optimizer, initially via a blog post and Twitter, designed for hidden layers in neural networks.

2025-02

Moonshot AI, in collaboration with UCLA, releases Moonlight, an MoE model trained with Muon, demonstrating its scalability for LLMs and ~2x computational efficiency over AdamW.

2025-08-19

Tinygrad 0.11 integrates Muon optimizer support.

2025-08-23

DeepSpeed officially integrates support for the Muon optimizer, becoming the first major distributed training framework to adopt it.

DeepSpeed Now Supports Muon Optimizer

⚡ 30-Second TL;DR

Key Points

🧠 Deep Insight

🔑 Enhanced Key Takeaways

🛠️ Technical Deep Dive

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline

📎 Sources (24)

👉Related Updates