Tag: #optimizer6 results
Gefen is a new drop-in replacement for the AdamW optimizer designed to significantly reduce memory overhead during LLM training. It aims to make large-scale training more accessible by optimizing memory usage by up to 8x.
DeepSpeed has officially integrated support for the Muon optimizer, a high-performance optimization algorithm gaining traction among frontier AI labs. This update allows developers to leverage Muon's efficiency within the established DeepSpeed ecosystem.
Fork of karpathy/autoresearch introduces Weber electrodynamic optimizer adapting learning rates via physics-inspired velocity and acceleration. Adds hardware entropy seeding from RTL-SDR, multi-provider AI agent harness with tools, and multi-GPU support. Merged community experiments improve baseline val/bpb from 0.9979 to 0.9697.
Rose is a new stateless PyTorch optimizer with ultra-low VRAM (less than 8-bit AdamW), fast convergence, and strong generalization, licensed Apache 2.0. Benchmarks on MNIST show competitive accuracy despite sometimes higher training loss but lower validation loss. Easy to use for ML training with minimal memory overhead.
AdamWClip extends the AdamW optimizer with adaptive gradient clipping, eliminating the need for manual threshold setting. It incurs no additional memory use and only marginal computational overhead. Preliminary experiments indicate it significantly outperforms AdamW with standard grad_norm clipping.
Muon optimizer gained traction in LLM training but sees little use beyond Transformers, like in ConvNets. Despite a Cifar-10 speed record at announcement, searches yield no results. Post speculates on scalability issues or overlooked papers.