Rollout-Level Advantage-Prioritized Experience Replay for GRPO

2026-06-03

AI Summary

This work addresses the instability and low sample efficiency in Generalized Reinforcement Learning with Policy Optimization (GRPO) for large language model post-training, which stem from rapid policy drift causing experience replay samples to become outdated. To mitigate this issue, the authors propose a rollout-level experience replay mechanism tailored for GRPO, integrating rollout-based storage and sampling, advantage-weighted prioritized replay, a maximum step-age (τ_max) eviction strategy, and the construction of fresh anchor batches. Experiments on three Qwen3-Base model scales across five mathematical benchmarks demonstrate that the proposed method significantly outperforms both standard GRPO and naive replay baselines, achieving an average accuracy gain of 4.35 percentage points and an improvement of 0.579 in the AES efficiency metric for the 4B model.

Abstract

Reinforcement learning from verifiable rewards with GRPO is a standard approach for post-training reasoning LLMs. It remains sample inefficient. Each rollout is used for a single gradient update and then discarded. Naive replay is not well suited in this setting because LLM policies drift quickly per gradient step. Stored rollouts therefore become stale and can destabilize training. We propose a rollout-level replay buffer for GRPO that stores and samples individual rollouts rather than whole groups. The buffer bounds staleness through age eviction. Any rollout older than tau_max training steps is removed. The buffer also preserves on-policy data via fresh-anchored composition. Each batch keeps its fresh on-policy rollouts and then concatenates replay rollouts drawn separately from the buffer. We prioritize replay by per-rollout advantage magnitude and recycle individual rollouts whose advantages are large. Across three Qwen3-Base scales on five math benchmarks, our method outperforms GRPO and naive replay baselines. Gains are positive at every scale and grow with model size. The largest gain is +4.35 pp on the five-benchmark average at 4B. Under an AES metric that jointly measures accuracy and token efficiency, the efficiency margin over GRPO is again largest at 4B, at +0.579.

Problem

• sample inefficiency
• policy drift
• experience replay
• staleness
• reinforcement learning

Innovation

• rollout-level replay
• advantage-prioritized sampling
• GRPO
• staleness control
• on-policy anchoring