Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning

2026-07-24

AI Summary

Asynchronous reinforcement learning often suffers from optimization instability or even policy collapse due to reliance on stale off-policy data. This work proposes Entropy-Scaled Trust Region (ESTR), a method that dynamically rescales importance weights by uncovering a systematic relationship between the natural scale of importance ratios and token entropy, thereby effectively distinguishing sampling noise from meaningful exploration. The mechanism enhances training stability without requiring additional forward passes or explicit version-switch detection. Evaluated on long-horizon agent tasks and mathematical reasoning benchmarks, ESTR outperforms existing asynchronous approaches, achieving a 2.6× speedup over synchronous GRPO while maintaining comparable accuracy and superior train–inference consistency.

Abstract

Asynchronous reinforcement learning (RL) accelerates large language model (LLM) post-training by overlapping rollout generation with policy optimization, but the resulting stale, off-policy data can destabilize optimization and ultimately cause policy collapse. Existing methods typically retain or discard tokens based solely on the magnitude of their importance ratios, applying the same threshold uniformly across token positions. In this work, we reveal that the natural scale of the importance ratio varies systematically with token entropy. Under asynchronous dynamics, this entropy-ratio scaling dictates two distinct phenomena: at low entropy, the inherent train-inference discrepancy is drastically amplified into substantial sampling noise; at high entropy, in-flight weight updates naturally induce pronounced, legitimate exploratory deviations. Consequently, magnitude-only correction inadvertently admits the amplified noise while strictly masking out the essential exploration triggered by in-flight updates. To address this, we propose the Entropy-Scaled Trust Region (ESTR), which scales each token's off-policy deviation by its local entropy, requiring no auxiliary forward passes or explicit version-switch detection. Across long-horizon agentic tasks and mathematical reasoning benchmarks, ESTR consistently outperforms existing asynchronous methods and achieves the best train-inference consistency. Compared with synchronous GRPO, ESTR attains comparable accuracy while improving training speed by $2.6\times$.

Problem

• asynchronous reinforcement learning
• off-policy data
• importance sampling ratio
• policy collapse
• entropy

Innovation

• Entropy-Scaled Trust Region
• asynchronous reinforcement learning
• off-policy correction
• importance ratio
• train-inference consistency