🤖 AI Summary
In policy-based reinforcement learning, redundant transitions arising from causal dependencies within rollouts can lead to gradient duplication, value network lag, and training instability. This work proposes a simple yet effective sampling strategy that randomly discards a fixed proportion (e.g., 25%) of transition samples while preserving the full reward signal, thereby breaking redundant structures and stabilizing PPO training. The method requires only a single additional sampling step and does not alter the core PPO algorithm or introduce new components. Experimental results across five standard environments demonstrate that this approach significantly improves training consistency—measured by KL divergence, policy entropy, and value estimation—while maintaining comparable returns.
📝 Abstract
Training a reinforcement learning agent on-policy means collecting fresh experience at every update, and that experience comes with a hidden problem. Each state in a rollout is the direct output of the previous one, causally chained together by the agent's own actions. Because of this, consecutive transitions are never truly independent. They carry overlapping information, and the gradient signal the network receives ends up far more repetitive than the batch size suggests. The same directions get reinforced over and over, the value network struggles to keep up as the policy shifts, and training becomes quietly unstable in ways that reward curves alone rarely reveal.
This paper asks whether that redundancy can simply be removed. We show that randomly dropping a fixed fraction of transitions from the rollout, at the right stage so the reward signal stays intact, is enough to break the repetitive gradient structure and stabilize training. The change is minimal: one sampling step, no new components, no modification to the core algorithm, and it works with any PPO implementation. Across five environments of increasing difficulty, CartPole-v1, Acrobot-v1, LunarLander-v2, HalfCheetah-v5, and Hopper-v5, the method matches vanilla PPO on reward while producing more consistent training dynamics across KL divergence, policy entropy, and value estimates. Dropping 25% of transitions turns out to be the sweet spot: enough to disrupt the redundancy, not enough to thin the batch.