On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training

2026-01-12

AI Summary

This study investigates whether supervised fine-tuning (SFT) and reinforcement learning (RL) can be decoupled in the post-training of large language models. Through theoretical analysis and controlled experiments based on Qwen3-0.6B, the work provides the first rigorous evidence that SFT and RL are inherently coupled, regardless of their alternation order: applying RL after SFT increases the SFT loss, while performing SFT after RL reduces the reward score. The findings reveal a bidirectional performance degradation mechanism, demonstrating an intrinsic coupling between cross-entropy loss and reward signals. This insight offers both theoretical grounding and empirical support for the design of effective post-training strategies.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Reinforcement Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Post-training of large language models routinely interleaves supervised fine-tuning (SFT) with reinforcement learning (RL). These two methods have different objectives: SFT minimizes the cross-entropy loss between model outputs and expert responses, while RL maximizes reward signals derived from human preferences or rule-based verifiers. Modern reasoning models have widely adopted the practice of alternating SFT and RL training. However, there is no theoretical account of whether they can be decoupled. We prove that decoupling is impossible in either order: (1) SFT-then-RL coupling: RL increases SFT loss under SFT optimality and (2) RL-then-SFT coupling: SFT lowers the reward achieved by RL. Experiments on Qwen3-0.6B confirm the predicted degradation, verifying that SFT and RL cannot be separated without loss of prior performance in the post-training

Problem

• supervised fine-tuning
• reinforcement learning
• post-training
• decoupling
• large language models

Innovation

• supervised fine-tuning
• reinforcement learning
• post-training
• non-decoupling
• large language models