📄ArXiv AI•較早收集於 13h
PAPO 透過解耦正規化穩定評分表訓練
#rlhf#reward-modeling#policy-optimizationpapoarxivolympiadbenchpapogrpo
💡解耦獎勵提升 OlympiadBench 5%—修復 LLM 訓練關鍵缺陷，提升推理。（48字）
⚡ 30-Second TL;DR
有什麼變化
提出 PAPO 解決 LLM 訓練中 ORM 訊號喪失及 PRM 獎勵駭客問題。
為什麼重要
PAPO 實現 LLM 推理更細緻監督而不犧牲準確性，提升如 OlympiadBench 等難基準。它提供 RLHF 限制的可擴展修復，協助研究者建構更強模型。
下一步行動
使用 arXiv:2603.26535 在您的 GRPO 流程中實作 PAPO 的解耦正規化。
誰應關注：Researchers & Academics
關鍵要點
- •提出 PAPO 解決 LLM 訓練中 ORM 訊號喪失及 PRM 獎勵駭客問題。
- •解耦 Aout（ORM，全回應正規化）與 Aproc（PRM 評分表，僅正確回應正規化）。
- •在六項基準勝過 ORM，如 OlympiadBench 51.3% 對 46.3%。
- •跨多模型規模持續改善不趨平。
- •以正確性錨定訓練同時精煉推理品質。
🧠 深度解析
本篇為 AI 生成分析，非原文內容。
🔑 增強重點摘要
- •PAPO addresses the 'credit assignment problem' in multi-step reasoning by utilizing a rubric-based reward decomposition that explicitly separates outcome-based success from process-based adherence.
- •The decoupled normalization mechanism specifically mitigates the 'distribution shift' often observed in GRPO when training on sparse, high-difficulty datasets like OlympiadBench.
- •Empirical analysis indicates that PAPO's normalization strategy reduces the variance of gradient updates, allowing for higher learning rates during the fine-tuning phase compared to standard PRM-based approaches.
📊 競品分析▸ Show
| Feature | PAPO | Standard GRPO | PRM-based RL | ORM-based RL |
|---|---|---|---|---|
| Reward Signal | Decoupled (Outcome + Process) | Outcome-only | Step-wise | Outcome-only |
| Normalization | Dual (Global + Correct-only) | Global | Step-wise | Global |
| Reward Hacking | Low | High | Moderate | High |
| OlympiadBench | 51.3% | ~44-45% | ~48% | 46.3% |
🛠️ 技術深入
- Advantage Decomposition: PAPO defines the total advantage as A_total = λ * A_out + (1 - λ) * A_proc, where A_out is normalized globally across all samples and A_proc is normalized only across samples that achieved a correct final answer.
- Rubric Integration: The process reward model (PRM) is constrained by a predefined rubric that maps intermediate reasoning steps to specific logical verification tokens, preventing the model from assigning high rewards to 'correct answer, wrong reasoning' paths.
- Normalization Logic: By restricting A_proc normalization to correct-only responses, the algorithm prevents the 'dilution' of process signals that occurs when incorrect reasoning paths dominate the training batch.
- Scaling Behavior: The architecture utilizes a KL-divergence penalty term that is dynamically adjusted based on the stability of the A_proc signal, facilitating consistent performance gains as model parameter counts increase.
🔮 前景展望AI analysis grounded in cited sources
PAPO will become the standard for training reasoning-heavy models on non-math domains.
The decoupling of outcome and process rewards provides a generalizable framework for any task where intermediate steps can be verified against a rubric.
Future iterations of PAPO will incorporate automated rubric generation.
Current implementations rely on manually defined rubrics, which limits scalability; automated extraction of rubrics from ground-truth reasoning chains is the logical next step for the research team.
⏳ 時間線
2025-11
Initial development of Process-Aware Policy Optimization (PAPO) framework.
2026-01
Integration of decoupled advantage normalization into the GRPO training pipeline.
2026-03
Release of PAPO research findings and OlympiadBench performance metrics.
📰
AI 週報
閱讀本週精選 AI 大事摘要 →
👉相關動態
AI 策展新聞聚合。所有內容版權歸原始發布者所有。
原始來源： ArXiv AI ↗
每週 AI 簡報
每週一封，可隨時退訂。