Deep Dive Into RL and OPD for LLM Training
Learn the math and code connecting OPD, GRPO, pretraining, and SFT in frontier LLM training.
30-Second TL;DR
What Changed
Explains the mathematical foundations of on-policy distillation for LLM training.
Why It Matters
The tutorial can help practitioners understand why post-training methods such as OPD and GRPO are increasingly important in frontier LLM development. It may also lower the barrier to reproducing or adapting these techniques in research codebases.
What To Do Next
Watch the tutorial and implement a small GRPO or OPD experiment on an open LLM to compare training stability, reward behavior, and downstream evaluation.
Key Points
- •Explains the mathematical foundations of on-policy distillation for LLM training.
- •Provides code-oriented coverage of GRPO-style reinforcement learning algorithms.
- •Relates RL and OPD methods to pretraining and supervised fine-tuning workflows.
- •Uses recent frontier-model reports from Kimi, DeepSeek, Qwen, and GLM as context.
Deep Insight
AI-generated analysis for this event — not the original article.
Enhanced Key Takeaways
- •GRPO (Group Relative Policy Optimization) eliminates the need for a separate critic model by estimating the baseline from group-wise rewards, significantly reducing VRAM requirements during RLHF.
- •On-Policy Distillation (OPD) is increasingly utilized to align smaller student models with the reasoning traces of larger 'reasoning' models, effectively transferring chain-of-thought capabilities without full-scale RL.
- •The shift toward 'test-time compute' scaling, popularized by DeepSeek-R1 and similar architectures, relies on these RL methods to optimize reasoning paths rather than just static output accuracy.
- •Recent implementations of GRPO often incorporate a KL-divergence penalty against a reference model to prevent policy collapse and maintain linguistic diversity during the reinforcement learning phase.
- •Integration of these techniques into standard SFT pipelines allows for 'iterative alignment,' where models are continuously refined through synthetic data generation and subsequent on-policy updates.
Technical Deep Dive
- GRPO Objective: Maximizes the expected reward by calculating the advantage of each output in a group relative to the group mean, defined as A_i = (r_i - mean(r)) / std(r).
- Memory Efficiency: By removing the Value Function (Critic) network, GRPO reduces the parameter count by approximately 50% compared to traditional PPO implementations.
- KL Penalty: A reference model is typically kept in memory to compute the KL divergence, ensuring the updated policy does not deviate excessively from the base SFT model.
- Reward Modeling: These methods often employ rule-based rewards (e.g., format verification, correctness) alongside model-based rewards to guide the reasoning process.
Future ImplicationsAI analysis grounded in cited sources
Timeline
- 2024-02DeepSeek releases early research on GRPO, demonstrating efficient RL for reasoning models.
- 2025-01DeepSeek-R1 launches, showcasing the practical application of GRPO at scale.
- 2025-06Qwen and GLM teams publish technical reports detailing the integration of OPD into their alignment pipelines.
- 2026-03Kimi (Moonshot AI) releases updated reasoning benchmarks utilizing refined on-policy distillation techniques.
Weekly AI Recap
Read this week's curated digest of top AI events →
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/MachineLearning ↗
This is a summary, not the original. Read the source, or get the weekly briefing.
The weekly digest
One email a week. Unsubscribe anytime.