Fix SFT issues before RL training

🔑 Enhanced Key Takeaways

•Supervised Fine-Tuning (SFT) is crucial for establishing foundational instruction-following and task-specific capabilities in large language models, serving as a necessary precursor to Reinforcement Learning from Human Feedback (RLHF) for refining nuanced preferences and safety.
•Multimodal SFT specifically teaches models to cross-reference and integrate information from diverse data types (text, images, audio, video), which is essential for addressing challenges like visual grounding and reducing hallucinations in Vision Language Models (VLMs) for domain-specific applications.
•Despite its importance, SFT can suffer from an 'Incomplete Learning Phenomenon' (ILP), where models fail to fully internalize parts of their training data due to issues such as missing prerequisite knowledge, conflicts with pre-training, internal data inconsistencies, or insufficient optimization for rare patterns.
•SFT, particularly in reasoning-focused Vision Language Models (VLMs), can sometimes induce 'pseudo reasoning paths' that are imitative and rigid, potentially degrading the performance of subsequent Reinforcement Learning (RL) stages, suggesting a need for careful SFT data quantity and quality.
•The creation of high-quality, diverse, and domain-specific SFT data is a significant bottleneck, being labor-intensive and costly, especially for complex alignment goals that require expert annotation and cover a vast range of potential inputs and edge cases.

🛠️ Technical Deep Dive

SFT involves adjusting a pre-trained model's internal weights and parameters by training on curated, labeled input-output pairs to minimize the error between predictions and desired outputs.
Key challenges in SFT include ensuring high data quality, preventing overfitting (especially with limited datasets), managing the scalability of data creation, and mitigating 'catastrophic forgetting' of knowledge acquired during pre-training.
For multimodal models, SFT extends to processing both text and image data within a unified training framework, enabling the model to learn complex relationships and cross-modal reasoning.
Techniques like Low-Rank Adaptation (LoRA) are being investigated to make SFT more parameter-efficient and to reduce catastrophic forgetting by minimizing changes to the base model's general knowledge.
Novel approaches are emerging, such as PRISM, a three-stage pipeline (SFT, Distribution Alignment via On-Policy Distillation, and Reinforcement Learning with Verifiable Rewards) designed to explicitly mitigate distributional drift introduced by SFT in large multimodal models.
Negative Supervised Fine-Tuning (nSFT) is a proposed memory-efficient alternative to multimodal RLHF, which leverages the 'negative supervision' (logits of rejected responses) found in RLHF paradigms and integrates it with a simpler SFT loss.

🔮 Future ImplicationsAI analysis grounded in cited sources

AI development will increasingly adopt hybrid training pipelines, starting with robust SFT and then applying RLHF or similar preference alignment techniques.

Many industry experts and research papers advocate for a combined SFT-then-RLHF approach as optimal for achieving both strong task performance and nuanced human alignment, with SFT providing a solid foundation and RLHF refining subjective preferences.

There will be a greater emphasis on advanced data curation, diagnostic tools, and pre-alignment stages for SFT to prevent 'incomplete learning' and 'pseudo reasoning paths'.

Ongoing research highlights persistent SFT failures like the Incomplete Learning Phenomenon (ILP) and the issue of SFT inducing imitative, rigid reasoning, necessitating more sophisticated data quality control, diagnostic frameworks, and intermediate alignment steps.

New SFT variants and optimization techniques will emerge to address the computational and data-intensive limitations of traditional SFT, especially in multimodal contexts.

Innovations such as Negative Supervised Fine-Tuning (nSFT) and parameter-efficient methods like LoRA indicate active development to make SFT more robust, memory-efficient, and scalable for complex multimodal models.

⏳ Timeline

2023-05

ChatGPT integrates unsupervised pre-training, SFT, instruction tuning, and RLHF for conversational abilities.

2023-11

Early works apply RLHF to LLMs for abstractive summarization and language model alignment.

2024-01

Multimodality in LLMs (e.g., ChatGPT's ability to 'see, hear, and speak') introduces new alignment challenges for SFT and RLHF data sourcing.

2024-11

Direct Preference Optimization (DPO) emerges as a more efficient alternative to the multi-stage process of RLHF.

2025-04

Research indicates that SFT can induce 'pseudo reasoning paths' that undermine subsequent RL performance in Vision-Language Models (VLMs).

2026-05

PRISM, a three-stage pipeline (SFT -> Distribution Alignment -> RLVR), is proposed to mitigate distributional drift introduced by SFT in Large Multimodal Models (LMMs).

Fix SFT issues before RL training

⚡ 30-Second TL;DR

Key Points

🧠 Deep Insight

🔑 Enhanced Key Takeaways

🛠️ Technical Deep Dive

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline

📎 Sources (23)

👉Related Updates