SourceReddit r/MachineLearning•Stalecollected in 31m
QLoRA 2e-4 learning rate is often too high
#fine-tuning#overfittingqloraqloraalpacaunsloth
Stop wasting weeks on bad fine-tuning runs; the standard QLoRA learning rate is likely ruining your small datasets.
30-Second TL;DR
What Changed
2e-4 is a legacy default derived from the Alpaca dataset (52k samples).
Why It Matters
Practitioners can save significant compute time and improve model quality by avoiding the 'overfitting trap' common in small-scale fine-tuning workflows.
What To Do Next
If fine-tuning on fewer than 10k samples, start your QLoRA learning rate at 1e-4 instead of the default 2e-4.
Who should care:Developers & AI Engineers
Key Points
- •2e-4 is a legacy default derived from the Alpaca dataset (52k samples).
- •Small datasets (under 10k) suffer from overfitting at 2e-4, leading to stagnant evaluation loss.
- •Lowering the learning rate to 1e-4 and increasing epochs often yields better results for small-scale fine-tuning.
- •Always tune the learning rate rather than relying on hardcoded defaults from tutorials.
Deep Insight
AI-generated analysis for this event — not the original article.
Enhanced Key Takeaways
- •The 2e-4 default originated from the original QLoRA paper (Dettmers et al., 2023), which utilized the Guanaco model trained on the OASST1 dataset, not exclusively the Alpaca dataset.
- •Learning rate sensitivity in QLoRA is heavily influenced by the rank (r) and alpha (lora_alpha) hyperparameters; higher ranks often require lower learning rates to maintain stability.
- •Weight decay settings are frequently overlooked in QLoRA fine-tuning; when lowering the learning rate, practitioners often need to adjust weight decay to prevent the model from collapsing into a suboptimal local minimum.
- •The '2e-4' convention is specifically tied to the use of 4-bit NormalFloat (NF4) quantization, which introduces a specific noise profile that higher learning rates can exacerbate if the dataset lacks sufficient diversity.
- •Recent empirical studies suggest that using a cosine learning rate scheduler with a significant warmup period (e.g., 10% of total steps) is more critical for convergence than the absolute peak learning rate value.
Technical Deep Dive
- QLoRA utilizes 4-bit NormalFloat (NF4) quantization to reduce memory footprint while maintaining performance comparable to 16-bit fine-tuning.
- The Double Quantization technique reduces memory usage by quantizing the quantization constants themselves.
- Paged Optimizers are employed to handle memory spikes during gradient updates by offloading to CPU RAM when GPU VRAM is exhausted.
- The interaction between the LoRA adapter rank (r) and the learning rate is non-linear; smaller ranks (e.g., r=8 or 16) are more susceptible to catastrophic forgetting at high learning rates compared to higher ranks (e.g., r=64 or 128).
Future ImplicationsAI analysis grounded in cited sources
Automated hyperparameter optimization (HPO) will replace manual learning rate selection in standard fine-tuning pipelines.
The high sensitivity of QLoRA to dataset size and rank makes manual tuning inefficient and prone to human error.
Standardized 'recipes' for fine-tuning will shift toward dataset-size-aware defaults.
As the community moves away from one-size-fits-all defaults, libraries like PEFT and Axolotl are increasingly adopting dynamic hyperparameter scaling based on input data volume.
Timeline
2023-05
QLoRA paper released by Tim Dettmers et al., introducing 4-bit quantization and the 2e-4 default.
2023-06
Integration of QLoRA into the Hugging Face PEFT library, standardizing the 2e-4 default in mainstream tutorials.
2024-02
Emergence of community-led 'fine-tuning guides' highlighting the limitations of default hyperparameters on small, high-quality datasets.
- 2023-05QLoRA paper released by Tim Dettmers et al., introducing 4-bit quantization and the 2e-4 default.
- 2023-06Integration of QLoRA into the Hugging Face PEFT library, standardizing the 2e-4 default in mainstream tutorials.
- 2024-02Emergence of community-led 'fine-tuning guides' highlighting the limitations of default hyperparameters on small, high-quality datasets.
Weekly AI Recap
Read this week's curated digest of top AI events →
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/MachineLearning ↗
This is a summary, not the original. Read the source, or get the weekly briefing.
The weekly digest
One email a week. Unsubscribe anytime.