Why LLMs Still Learn Mostly by Imitation

Post LinkedIn

🧐Read original on LessWrong AI

#imitative-learning #chain-of-thought #model-alignmentllm-trainingllms rlvr rlhf sft

💡A useful reality check on whether RLVR—not pretraining—is driving today’s LLM capabilities.

⚡ 30-Second TL;DR

What Changed

Imitative learning includes both large-scale pretraining and supervised fine-tuning, forming the primary foundation of LLM capabilities.

Why It Matters

The analysis cautions teams against overstating RLVR’s role when evaluating where frontier-model capabilities originate. It also suggests that training-budget decisions and alignment research should account for the dominant influence of pretrained and supervised data.

What To Do Next

Run matched-compute ablations comparing pretraining or SFT against RLVR to measure each method’s effect on your model’s capabilities and reasoning behavior.

Who should care:Researchers & Academics

Key Points

•Imitative learning includes both large-scale pretraining and supervised fine-tuning, forming the primary foundation of LLM capabilities.
•The article cites estimates that even if RLVR uses 50% of training compute, it may provide only about 0.01% of the information imparted during training.
•Human-created text continues to strongly influence model chain-of-thought, even when reasoning traces appear strange or partially illegible.
•The distinction between imitative learning and reinforcement learning affects how practitioners interpret chain-of-thought legibility, capabilities, and alignment.

🧠 Deep Insight

AI-generated analysis for this event.

🔑 Enhanced Key Takeaways

•Information Bottleneck Theory suggests that during pretraining, LLMs prioritize compressing the vast statistical distribution of human language, which inherently limits the 'reasoning' gains achievable through sparse reward signals in RL.
•Recent research into 'Process Reward Models' (PRMs) indicates that while they improve step-by-step verification, they often rely on 'imitation of human-labeled reasoning paths' rather than discovering novel, non-human-like problem-solving strategies.
•The 'Data Scaling Law' hypothesis posits that the diminishing returns of RL are due to the 'quality ceiling' of synthetic data generated by models, which often collapses into mode-seeking behavior without fresh, high-entropy human input.
•Emergent reasoning capabilities in models like o1 and its successors are increasingly attributed to 'Test-Time Compute' (inference-time search) rather than the reinforcement learning phase itself, effectively decoupling reasoning from the training objective.
•Cross-entropy loss, the primary objective of imitative learning, remains the most efficient way to maximize the 'bits per parameter' metric, whereas RL-based objectives often introduce noise that degrades the model's foundational linguistic knowledge.

🛠️ Technical Deep Dive

Pretraining Objective: Primarily Autoregressive Next-Token Prediction (NTP) using Cross-Entropy Loss on massive corpora.
RLVR (Reinforcement Learning from Verifiable Rewards): Utilizes sparse feedback loops where models are rewarded for correct final answers, often failing to optimize the latent reasoning process.
Test-Time Compute: Shift from training-time optimization to inference-time search (e.g., Monte Carlo Tree Search or Best-of-N sampling) to enhance reasoning without altering model weights.
Information Density: Analysis of KL-divergence between pretrained models and RL-tuned models shows that RL often results in 'policy drift' where the model loses general-purpose capabilities to satisfy specific reward constraints.

🔮 Future ImplicationsAI analysis grounded in cited sources

Pretraining will remain the dominant driver of capability growth through 2027.

The massive disparity in information density between imitative learning and RL suggests that scaling compute for pretraining remains more cost-effective than scaling RL for reasoning.

Test-time compute will replace RL as the primary method for improving reasoning.

Inference-time search allows models to explore reasoning paths without the catastrophic forgetting or reward hacking associated with reinforcement learning.

⏳ Timeline

2020-05

GPT-3 release demonstrates the power of massive-scale imitative learning.

2022-11

ChatGPT launch popularizes RLHF as a method for alignment, though capabilities remain rooted in pretraining.

2024-09

Introduction of reasoning-focused models (e.g., o1-preview) shifts industry focus toward test-time compute.

2025-06

Research papers confirm that RLVR performance gains are highly dependent on the quality of the underlying imitative base model.

🧐Read original article on LessWrong AI

📰

Weekly AI Recap

Read this week's curated digest of top AI events →

👉Related Updates

Same topic

Explore #imitative-learning

Same product