Rethinking the Divergence Regularization in LLM RL

2026-06-08

AI Summary

This work addresses the susceptibility of large language models to distributional shift in off-policy reinforcement learning, where existing approaches relying on importance sampling or hard clipping struggle to maintain stable updates under long-tailed vocabularies. The paper proposes DRPO, a novel method that refines the conventional hard-clipped trust-region mechanism by introducing a smooth, advantage-weighted quadratic regularizer based on KL divergence. This formulation preserves the geometric structure of DPPO while imposing continuous and bounded gradient weights on policy deviations, effectively mitigating divergence and providing corrective signals beyond policy boundaries. Empirical results demonstrate that DRPO significantly enhances training stability and sample efficiency across diverse model scales, architectures, and precision settings.

Abstract

Reinforcement learning (RL) has become a key component of post-training large language models (LLMs). In practice, LLM RL is often off-policy because of training-inference mismatch and policy staleness, making trust-region control essential for stable optimization. Mainstream methods such as PPO and GRPO approximate this control with a ratio-clipping mechanism, but the importance ratio can be a poor proxy for distributional shift in long-tailed vocabularies. Recent work such as DPPO addresses this mismatch by replacing ratio-based clipping with a divergence-based mask, yielding a trust region defined by the sampled token's absolute probability shift. However, DPPO still relies on a hard mask: once a token crosses the trust-region boundary in a harmful direction, its gradient is discarded rather than corrected. To address this, we propose Divergence Regularized Policy Optimization (DRPO), which replaces the hard mask with a smooth advantage-weighted quadratic regularizer on policy shift. DRPO preserves the same trust-region geometry as DPPO while inducing bounded, continuous gradient weights that attenuate diverging updates and provide corrective signals beyond the boundary. Experiments across model scales, architectures, and precision settings show that DRPO improves the stability and efficiency of LLM RL training.

Problem

• trust-region control
• distributional shift
• divergence regularization
• off-policy RL
• large language models

Innovation

• Divergence Regularization
• Trust Region
• Policy Optimization
• Large Language Models
• Reinforcement Learning