🤖Reddit r/MachineLearning•Stalecollected in 56h
Seeking NTK Lazy vs Rich Learning Resources
#ntk#lazy-training#rich-regimes#initializationntk-lazy-rich-regimesntk
💡Unlock NTK insights for lazy/rich regimes—key to modern NN training
⚡ 30-Second TL;DR
What Changed
Heuristics to detect lazy (static NTK) vs rich (feature learning) regimes during training.
Why It Matters
Guides practitioners tuning models between lazy kernel and rich feature learning for better performance.
What To Do Next
Search arXiv for 'Neural Tangent Kernel lazy rich regimes' to find key papers like Dyer & Gur-Ari.
Who should care:Researchers & Academics
Key Points
- •Heuristics to detect lazy (static NTK) vs rich (feature learning) regimes during training.
- •How init scale and LR bias toward feature learning over kernel regime.
- •Architectures preferring lazy regime for stability; richness as spectrum.
🧠 Deep Insight
Background and context from public sources — not the original article. 7 sources cited.
🔑 Enhanced Key Takeaways
- •Grokking represents a transitional phenomenon between lazy (NTK-dominated) and rich (feature-learning) regimes, where networks exhibit delayed generalization before suddenly improving—preconditioned gradient descent can accelerate this transition by enabling uniform exploration of the NTK subspace without spectral bias[2].
- •Exact analytical solutions for learning dynamics in deep linear networks under lambda-balanced initializations reveal how weight scaling across layers determines the spectrum between lazy and rich regimes, with direct implications for continual learning, reversal learning, and transfer learning[1][3].
- •Adaptive kernel predictors derived from feature-learning infinite-width limits outperform traditional lazy-regime kernels on benchmark datasets, demonstrating that rich-regime networks converge to data-dependent kernel machines rather than fixed kernels[6].
- •Spectral bias in the lazy regime causes slow, non-uniform exploration of frequency modes during training; higher-order gradient methods (like preconditioned gradient descent) mitigate this by accelerating convergence across all frequency components[2].
- •The rich regime requires small initialization scales (small α) where weight changes substantially impact outputs, necessitating learning rate scaling as η ∼ α⁻² to prevent gradient explosion or vanishing, whereas the lazy limit uses large α with minimal feature learning[5].
🛠️ Technical Deep Dive
- •Lambda-balanced initializations parameterize weight scaling across network layers; exact gradient flow solutions exist for two-layer linear networks, capturing representation evolution and Neural Tangent Kernel dynamics across the lazy-rich spectrum[1][3].
- •Preconditioned gradient descent (PGD) reduces spectral bias by enabling uniform exploration of the NTK subspace, eliminating characteristic grokking delays and accelerating convergence to all frequency modes in the lazy regime[2].
- •Feature learning in the rich regime manifests as high-rank adjustments to the tangent kernel's eigenstructure, with kernel alignment to challenging label kernels accelerating learning and capturing residual variance beyond final-layer adaptation[4].
- •Dynamical mean field theory (DMFT) applied to infinite-width gradient flow with weight decay yields fixed-point equations defining task-adapted internal representations and adaptive kernel predictors that outperform NTK-based kernels[6].
- •Lazy training preserves the tangent feature space at initialization; functional improvement is limited to projections already present, with no feature selection or alignment to data distribution occurring[4].
🔮 Future ImplicationsAI analysis grounded in cited sources
Adaptive kernels from feature-learning limits will become standard for high-performance neural network design, replacing fixed NTK approaches in practical applications.
Empirical results show adaptive kernels achieve lower test loss than lazy-regime kernels on benchmark datasets, suggesting a shift toward exploiting rich-regime dynamics in production systems[6].
Preconditioned optimization methods will accelerate grokking phenomena in large-scale models, reducing training time for sudden generalization jumps.
PGD eliminates spectral bias delays by enabling uniform NTK subspace exploration, providing a direct mechanism to speed up the lazy-to-rich transition observed in language and vision models[2].
Initialization heuristics based on lambda-balanced scaling will become standard practice for controlling lazy vs. rich regime selection, enabling practitioners to tune feature learning intensity.
⏳ Timeline
2018-06
Neural Tangent Kernel framework introduced by Jacot et al., establishing theoretical foundation for lazy training regime analysis
2020-01
Feature learning and rich regime concepts formalized in deep learning theory, contrasting with NTK lazy limit
2023-01
Gradient Feature Learning Framework published, deriving generalization guarantees for SGD-trained networks and explaining lottery ticket hypothesis
2024-06
Grokking identified as transitional behavior between lazy NTK regime and rich feature-learning regime by Kumar et al. and Zhou et al.
2025-02
Preconditioned Gradient Descent research demonstrates acceleration of grokking and uniform NTK subspace exploration without spectral bias
2025-05
Adaptive kernel predictors from feature-learning infinite limits published, showing superior performance over lazy-regime kernels on benchmarks
📎 Sources (7)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
📰
Weekly AI Recap
Read this week's curated digest of top AI events →
👉Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/MachineLearning ↗
This is a summary, not the original. Read the source, or get the weekly briefing.
Weekly AI briefing
One email a week. Unsubscribe anytime.