ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Information

2026-06-01

AI Summary

This work addresses the challenge of distributional shift in asynchronous reinforcement learning for large language model post-training, which arises from the use of stale responses and is exacerbated by existing methods' reliance on behavior policy probabilities—quantities difficult to maintain in real-world systems. To overcome this limitation, the paper proposes ASymPO, an asynchronous population-based relative reinforcement learning method that requires only the current policy probabilities. By normalizing token-level losses per response, ASymPO restores response-level zero-sum loss balance without access to behavior policy information for the first time. Furthermore, it introduces SPO as a fixed negative scaling baseline to mitigate scale imbalance between positive and negative loss terms. Experiments demonstrate that ASymPO enables stable optimization in asynchronous mathematical reasoning post-training while preserving effective non-zero learning signals.

Abstract

Asynchronous reinforcement learning can improve language-model post-training throughput by decoupling response generation from policy optimization, but stale responses introduce distribution drift. Standard behavior-corrected methods control this drift with behavior-policy probabilities, importance ratios, or clipping, which requires token-aligned, versioned, and numerically consistent behavior log-probabilities across rollout and learner systems. We ask whether asynchronous group-relative RL can instead be stabilized using only current-policy probabilities. We identify a scale-imbalance failure mode: when stale responses are evaluated under the current policy, positive and negative loss terms can appear at different negative log-probability scales, so zero-sum advantages no longer imply balanced loss contributions. We propose Asymmetric-Scale Policy Optimization (ASymPO), which normalizes each response's token loss by its current average token negative log-probability. ASymPO requires no behavior-policy probabilities, restores response-level zero-sum balance, and preserves a nonzero learning signal. We also introduce Scaled Policy Optimization (SPO), a fixed negative-scaling baseline, and evaluate both current-policy-only objectives in asynchronous mathematical reasoning post-training.

Problem

• asynchronous reinforcement learning
• distribution drift
• behavior-policy probabilities
• language model post-training
• policy optimization

Innovation

• Asymmetric-Scale Policy Optimization
• asynchronous reinforcement learning
• behavior-free policy optimization
• distribution drift
• large language model post-training