Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directions

2026-09-21

AI Summary

本文针对推理导向强化学习中的rollout效率问题，通过系统分类现有方法并分析其瓶颈，提出提高效率的策略及未来研究方向。

Abstract

Reasoning-oriented reinforcement learning enables large language models to solve mathematical, coding, and other multi-step tasks, but shifts a substantial portion of the training cost to rollout, where trajectories are generated for policy updates. Efficient rollout mechanisms are therefore essential to reduce this cost while maintaining the freshness, consistency, and statistical validity of training data. This survey provides a systematic taxonomy of recent research on rollout efficiency for reasoning-oriented reinforcement learning, classifying existing approaches from both mechanism and bottleneck perspectives. Based on this taxonomy, we analyze how different technique families address distinct sources of rollout inefficiency, examine opportunities and potential conflicts for combining them, identify gaps in the evaluation and reporting of efficiency gains, and discuss open challenges and future research directions.

Problem

• Rollout Efficiency
• Reinforcement Learning
• Reasoning
• Large Language Models
• Training Cost

Innovation

• rollout efficiency
• reasoning-oriented reinforcement learning
• systematic taxonomy
• training cost reduction
• data freshness