ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learning

2026-07-27

AI Summary

This work addresses the instability commonly observed in reinforcement learning training with large language models, which arises from architectural mismatches and precision discrepancies between training and inference—such as FP8 inference versus higher-precision training. To mitigate this issue, the paper proposes Adaptive Control Reinforcement Learning (ACRL), a method that dynamically regulates the divergence between training and inference within a bounded range, thereby stabilizing the learning process. Notably, this mechanism naturally enhances policy entropy without requiring explicit entropy regularization, leading to improved exploration efficiency. Experimental results demonstrate that under FP8 low-precision inference, ACRL achieves significantly better training stability compared to importance sampling correction methods and attains final performance on par with BF16 high-precision baselines.

Abstract

Reinforcement Learning (RL) training for Large Language Models (LLMs) often suffers from instability due to the discrepancy between training and inference. This training-inference discrepancy stems from two primary factors: an architectural separation between training and inference engines, and the use of low-precision quantization in inference versus higher-precision computation in training. To address training instability issues caused by high training-inference discrepancy, we present the principles and methods for its adaptive control. We propose Adaptive Control Reinforcement Learning (ACRL), which adaptively maintains the training-inference discrepancy within a reasonable range to ensure stable RL training. Beyond stabilization, ACRL inherently increases policy entropy, thereby enhancing exploration and improving accuracy. The experimental results show that when the inference engine utilizes FP8 quantization, ACRL consistently maintains the training-inference discrepancy within a reasonable range and stabilizes RL training. Furthermore, ACRL not only matches the accuracy of the BF16 baseline but also outperforms importance sampling (IS) fixes.

Problem

• training-inference discrepancy
• reinforcement learning
• Large Language Models
• quantization
• training instability

Innovation

• Adaptive Control
• Training-Inference Discrepancy
• Reinforcement Learning
• Quantization
• Policy Entropy