AReaL-DTE: Sparse Policy-Weight Transfer for Online Agentic Reinforcement Learning

2026-08-01

AI Summary

This work addresses the high model transmission and deployment overhead in online reinforcement learning caused by frequent dense policy synchronization. The authors propose a snapshot-free incremental synchronization engine that introduces a novel sparse weight transfer mechanism, eliminating the need for full intermediate models. By leveraging inference-visible weight sparsity, the method employs BF16 change detection, coordinate remapping, and AdamW inverse updates to reconstruct overwritten parameters on demand. Efficient end-to-end synchronization is achieved through a streaming converter alignment and manifest commit protocol. The approach supports cross-cluster shared storage and a deadlock-free two-phase protocol, demonstrating up to 19.9× speedup across clusters and 7.6× within a cluster on Qwen3-8B and Qwen3-30B-A3B models, while reducing peak GPU and CPU memory usage within a cluster by over 41% and 87%, respectively.

Abstract

Online agentic reinforcement learning implemented with micro-services separates policy training from rollout generation, improving scalability and modularity while potentially making frequent policy-weight synchronization a critical systems overhead. Shared storage naturally connects these services across clusters, but vanilla dense policy weight synchronization could incur model-scale construction, transfer, and application costs. Sparse synchronization reduces transferred data, yet checkpoint-oriented approaches can still retain a previous model and materialize complete intermediates to bridge heterogeneous training and inference layouts. We present AReaL-DTE, a snapshot-free Delta Transfer Engine that translates inference-visible weight sparsity into end-to-end system efficiency. Across our evaluated workloads, fewer than 2% of BF16 weight elements change between consecutive policy versions. AReaL-DTE reconstructs overwritten weights on demand by inverting AdamW updates, streams reconstructed and current parameters through converter-aligned BF16 change detection, and remaps changed elements directly into receiver-local coordinates. AReaL-DTE supports manifest-committed sparse transfer through shared storage across clusters and a deadlock-safe two-round protocol within a cluster, followed by direct application to inference shards. We evaluate AReaL-DTE on Qwen3-8B and Qwen3-30B-A3B across four online RL workloads. AReaL-DTE achieves speedups of up to 19.9x over ByteCheckpoint and 3.2x over PULSE across clusters, and up to 7.6x and 7.4x, respectively, within a cluster. In the same-cluster Qwen3-30B-A3B experiments, it reduces peak GPU memory by approximately 41% and peak CPU memory by at least 87%.

Problem

• online reinforcement learning
• policy-weight synchronization
• sparse transfer
• micro-services
• system overhead

Innovation

• Sparse Policy Transfer
• Delta Synchronization
• Online Reinforcement Learning
• Memory-Efficient Inference
• Distributed Microservices