GRPO Scaling Produces Surprisingly Uneven Results
💡The middle-sized model degraded most under identical GRPO settings—revealing why scale alone cannot predict post-trainin
⚡ 30-Second TL;DR
What Changed
WikiText perplexity worsened after SFT-to-GRPO by 0.2% on V1, 52% on V2, and 5% on V3.
Why It Matters
The results suggest that GRPO post-training quality may depend more on policy format, reward design, and optimization stability than on parameter count alone. Practitioners should avoid assuming that successful reward learning implies broad reasoning transfer.
What To Do Next
Rerun the GRPO comparison with matched chat templates, an explicit stop or length penalty, and KL-coefficient sweeps before scaling the recipe to production models.
Key Points
- •WikiText perplexity worsened after SFT-to-GRPO by 0.2% on V1, 52% on V2, and 5% on V3.
- •The 316M V2 model degraded the most, challenging the assumption that larger models are always more vulnerable or more robust.
- •V3 mastered four of five arithmetic curriculum stages, but GSM8K performance remained near zero.
- •The experiment was confounded by changes in model architecture, data mix, token count, formatting, and the absence of a stopping reward.
🧠 Deep Insight
AI-generated analysis for this event.
🔑 Enhanced Key Takeaways
- •GRPO (Group Relative Policy Optimization) relies on generating multiple outputs per prompt to estimate a baseline, which often fails in sub-1B parameter models due to high variance in policy gradients.
- •The observed degradation in perplexity is frequently linked to 'reward hacking' where small models collapse their probability distribution toward a narrow set of tokens that satisfy the reward function but lose general linguistic capability.
- •Curriculum learning in GRPO often requires a 'warm-up' phase with standard SFT that exceeds the token counts used in this experiment, suggesting the models were under-trained for the complexity of the task.
- •The lack of a stopping reward (or EOS token penalty) in the experiment likely caused the models to generate excessively long, repetitive sequences, further skewing the perplexity metrics.
- •Recent research indicates that GRPO performance is highly sensitive to the 'group size' parameter; smaller models often require larger group sizes to stabilize the baseline, which was likely constrained by the hardware used in this study.
🛠️ Technical Deep Dive
- GRPO eliminates the need for a separate Value Function (Critic) model, reducing memory overhead but increasing the computational cost of the forward pass during training.
- The algorithm calculates the advantage by comparing each output in a group to the mean reward of that group, making it highly dependent on the diversity of the generated samples.
- Small models (sub-1B parameters) often suffer from 'mode collapse' during GRPO because they lack the capacity to maintain diverse reasoning paths when the reward signal is sparse.
- The absence of a KL-divergence penalty against a reference model in some custom GRPO implementations often leads to rapid divergence from the base language model distribution.
🔮 Future ImplicationsAI analysis grounded in cited sources
Weekly AI Recap
Read this week's curated digest of top AI events →
👉Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/MachineLearning ↗