Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning

2026-07-24

AI Summary

This work addresses the inefficiency and instability of fixed rollout allocation in reinforcement learning for long-chain reasoning tasks, where many rollouts carry insufficient information. The authors propose VIGOR, a novel method that, for the first time, theoretically links group reward variance to gradient magnitude and establishes its acceleration ratio over GRPO. VIGOR introduces a variance-guided online dynamic rollout allocation mechanism: it initially assigns a small number of rollouts and iteratively adds more based on observed variance until the budget is exhausted. Integrated within the RLVR framework and leveraging Pareto distribution modeling with a progressive policy strategy, VIGOR reduces the required rollouts to reach target accuracy by a factor of 2.3 compared to GRPO on mathematical reasoning tasks, achieves the same pass@k rate in code generation with 1.49× fewer rollouts, and improves average test pass rates by 3.4 percentage points.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as a highly effective framework for improving LLM reasoning, with methods such as GRPO among its most successful instantiations. However, GRPO relies on repeated generation of long chain-of-thought rollouts. Training time scales with the number of rollouts, a large fraction of which are uninformative. Thus, GRPO is computationally expensive and unstable. To mitigate this, existing approaches either generate a larger pool of rollouts and filter the most informative prompts, or leverage historical signals for filtering at later stages of training. These strategies offer modest performance gains, but slow down the overall process. To address this, we propose VarIance Guided Online Rollout allocation (VIGOR) which instead of allocating a fixed rollout budget per example, begins with a small number of rollouts for all examples in a batch and iteratively allocates additional rollouts to those with the highest group reward variance until a fixed total rollout budget is reached. Theoretically, we show that under RLVR, reward variance controls the gradient magnitude, and derive VIGOR's closed-form speedup ratio over GRPO, which grows with refinement rounds under Pareto-distributed reward variance. Experiments on mathematical reasoning and coding tasks show that VIGOR reaches target accuracy with up to 2.3$\times$ fewer rollouts on math, reaches GRPO's final coding full pass rate with 1.49$\times$ fewer rollouts, and improves the coding average test pass rate by 3.4 points.

Problem

• reinforcement learning
• rollout allocation
• reward variance
• computational efficiency
• LLM reasoning

Innovation

• VIGOR
• rollout allocation
• reward variance
• reinforcement learning with verifiable rewards
• efficient LLM reasoning