When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff

2026-06-07

AI Summary

This work addresses the detrimental effects of overfitting during supervised fine-tuning (SFT), which can severely impair a model’s plasticity in subsequent reinforcement learning (RL) stages, manifesting as overconfident outputs and sharp minima in the parameter space. The study systematically uncovers, for the first time, the underlying mechanisms by which SFT overfitting hinders RL optimization. To mitigate this issue, the authors propose Rejuvenation—a novel strategy that combines base-model anchoring with targeted neuron resetting. This approach preserves the beneficial priors acquired during SFT while effectively restoring model plasticity. Empirical results demonstrate that Rejuvenation substantially enhances RL performance on math reasoning and agent-based tasks for models previously overfit during SFT, and further improves generalization to out-of-distribution tasks.

Abstract

Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) has become a standard pipeline for Large Language Model (LLM) post-training. SFT is expected to provide a useful behavioral prior for RL to further enhance model capabilities. However, checkpoints with excessive SFT often show limited improvement during RL. We attribute this failure to the loss of model plasticity: the reduced ability of an SFT-initialized policy to be effectively reshaped by subsequent RL. To better understand this phenomenon, we conduct detailed analysis from multiple perspectives, including parameter changes, output spaces, and RL optimization dynamics. Our results show that models from excessive SFT tend to produce over-confident token distributions and exhibit sharp parameter landscapes, which make them harder to optimize in the RL stage. To enable a more robust SFT-to-RL handoff, we propose \texttt{Rejuvenation}, a simple yet effective method that restores plasticity while preserving useful SFT-acquired priors. Rejuvenation leverages base-anchored model fusion to reduce excessive SFT-induced drift with targeted neuron reset to mitigate model rigidity. Experimental results on both math reasoning tasks and agentic tasks demonstrate that our approach consistently improves RL performance on over-trained SFT models, while also enhancing generalization to out-of-distribution tasks.

Problem

• model plasticity
• supervised fine-tuning
• reinforcement learning
• SFT-to-RL handoff
• overfitting

Innovation

• model plasticity
• SFT-to-RL handoff
• Rejuvenation
• base-anchored fusion
• neuron reset