Unleashing Efficient Asynchronous RL Post-Training via Staleness-Constrained Rollout Coordination

2026-01-19

AI Summary

This work addresses the challenges of stale trajectory data and skewed sequence lengths in asynchronous reinforcement learning post-training, which often degrade convergence and imbalance system throughput. To this end, we propose StaleFlow, the first system that jointly mitigates these issues by enforcing global consistency protocols to bound data staleness, adopting a decoupled data server architecture that separates trajectory storage from model parameters, and introducing a staleness-aware, high-throughput rollout coordination mechanism. Experimental results demonstrate that StaleFlow achieves 1.17–2.68× higher throughput compared to existing systems while preserving model convergence guarantees.

Technology Category

• Multiagent Systems: Multiagent Learning
• Machine Learning: Reinforcement Learning
• Search and Optimization: Learning to Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Reinforcement learning (RL) post-training has become pivotal for enhancing the capabilities of modern large models. A recent trend is to develop RL systems with a fully disaggregated architecture, which decouples the three RL phases (rollout, reward, and training) onto separate resources and executes them asynchronously. However, two critical data-level concerns arise: (1) asynchronous execution leads to data staleness in trajectories (the data generated by rollout) as the model parameters used in rollout may not be up to date, which impairs RL convergence; and (2) the length variation of trajectories introduces severe data skewness, leading to workload imbalance and degraded system performance. Existing systems fail to address these two concerns in a unified manner. Techniques that tightly control data staleness often constrain effective data skewness mitigation, while aggressive data skewness mitigation tends to exacerbate data staleness. As a result, systems are forced to trade off convergence for performance, or vice versa. To address this, we propose StaleFlow, an RL post-training system that jointly tackles data staleness and skewness. First, to control staleness, StaleFlow introduces a global consistency protocol that tracks the full lifecycle of each trajectory and constrains staleness. Second, to mitigate skewness, StaleFlow re-designs the RL system architecture by constructing data servers for trajectories and parameters to achieve flexible rollout coordination. Subsequently, we develop a suite of staleness-aware, throughput-oriented strategies to enhance system performance. Evaluations show that StaleFlow achieves up to 1.42-2.68$\times$ (1.17-2.01$\times$ on average) higher throughput than state-of-the-art systems, without compromising convergence.

Problem

• asynchronous reinforcement learning
• data staleness
• data skewness
• rollout coordination
• RL post-training

Innovation

• asynchronous reinforcement learning
• data staleness
• data skewness
• rollout coordination
• post-training