Score Centering Stabilizes Off-policy Reinforcement Learning

2026-09-17

AI Summary

该研究针对训练-推理不匹配导致的强化学习不稳定问题，提出了一种加法“分数中心化”校正方法来消除偏差，提高模型稳定性。

Abstract

Reinforcement learning (RL) of large language models is notoriously sensitive to small differences between training and inference engines, often referred to as the training-inference mismatch (TIM). However, completely eliminating TIM is impractical, as it would come at a major cost to rollout efficiency. In this paper, we show that the instability of RL under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step. We derive an additive "score centering" correction term that stabilizes RL under TIM by canceling drift. When training models from 0.6B to 30B parameters, score centering alone matches or outperforms methods based on importance sampling under quantization, with the gap growing as the mismatch becomes more severe. Because the correction is additive, score centering also composes with importance sampling -- their composition outperforms pure importance-sampling baselines in our staleness experiments.

Problem

• Reinforcement Learning
• Training-Inference Mismatch
• Drift

Innovation

• score centering
• training-inference mismatch
• drift cancellation
• off-policy reinforcement learning