Why Adam Loses GD’s Low-Rank Bias

Post LinkedIn

🤖Read original on Reddit r/MachineLearning

#optimizer-dynamics #low-rank-learning #matrix-sensing #implicit-biasloss-basis-adam-studyadam rmsprop muon shampoo

💡A controlled optimizer comparison links Adam’s coordinate-wise scaling to the loss of low-rank recovery bias.

⚡ 30-Second TL;DR

What Changed

Factored models W = UV^T are invariant under rotations of the factor basis, and standard GD respects this symmetry.

Why It Matters

The findings suggest that optimizer choice can alter the inductive bias of matrix-factorized and deep-linear systems even when training loss is matched. Practitioners using adaptive optimizers for low-rank learning should evaluate recovery quality and basis sensitivity rather than relying only on training or validation loss.

What To Do Next

Run the released matrix-sensing code and benchmark Adam against shared-scalar Adam with global norm clipping on your low-rank workload.

Who should care:Researchers & Academics

Key Points

•Factored models W = UV^T are invariant under rotations of the factor basis, and standard GD respects this symmetry.
•Per-coordinate adaptive denominators in Adam appear to introduce basis-dependent anisotropy that harms implicit low-rank recovery.
•A one-parameter transition from coordinate-wise to shared-scalar normalization improved recovery monotonically, isolating anisotropy as the likely mechanism.
•Muon matched GD on truly low-rank targets but degraded rapidly with spectral-tail energy, crossing over to GD near 4% tail energy.
•Replacing per-coordinate clipping with global norm clipping improved the author’s optimizer’s recovery error from 0.347 to 0.220.

🧠 Deep Insight

AI-generated analysis for this event.

🔑 Enhanced Key Takeaways

•The study highlights that adaptive optimizers like Adam effectively perform a coordinate-wise preconditioner that breaks the rotational symmetry inherent in matrix factorization, which is crucial for the implicit regularization observed in deep learning.
•Research indicates that the 'low-rank bias' is not merely a property of the loss landscape but is actively preserved by optimizers that maintain isotropic updates, such as GD and Shampoo.
•The transition from coordinate-wise to shared-scalar normalization suggests that the loss of low-rank bias is directly correlated with the degree of anisotropy introduced by the optimizer's preconditioner.
•Muon's performance degradation relative to GD in the presence of spectral-tail energy suggests that while it excels at capturing dominant singular vectors, it struggles with the noise or 'tail' components that standard GD handles more gracefully.
•The findings suggest that for tasks where low-rank structure is a primary inductive bias (e.g., certain types of representation learning or compression), adaptive optimizers may be suboptimal compared to non-adaptive or globally-scaled alternatives.

📊 Competitor Analysis▸ Show

Optimizer	Adaptive Type	Low-Rank Bias Preservation	Primary Use Case
GD	None	High	Theoretical analysis, small-scale tasks
Adam	Per-coordinate	Low	General purpose, LLM pre-training
Muon	Momentum-based	High (on low-rank)	Large-scale model training
Shampoo	Preconditioned	High	Large-scale, high-memory training
Lion	Sign-based	Low	Memory-efficient training

🛠️ Technical Deep Dive

The study utilizes underdetermined matrix-sensing tasks where the target matrix M = UV^T is recovered from linear measurements.
The anisotropy metric is quantified by the variance of the diagonal elements of the preconditioner matrix; higher variance correlates with lower rank-recovery performance.
Global norm clipping is identified as a critical intervention to mitigate the bias-breaking effects of per-coordinate clipping in adaptive optimizers.
The spectral-tail energy is defined as the sum of squared singular values beyond the rank-k threshold, serving as a proxy for the 'difficulty' of the recovery task.

🔮 Future ImplicationsAI analysis grounded in cited sources

Adaptive optimizers will incorporate 'rank-aware' scaling mechanisms.

The identified trade-off between convergence speed and implicit bias will drive the development of optimizers that switch from adaptive to isotropic scaling as training progresses.

Standard Adam will be replaced by shared-scalar variants in low-rank sensitive architectures.

Evidence that per-coordinate scaling harms low-rank recovery will necessitate the adoption of more isotropic alternatives in specialized model training pipelines.