Process Reward Informed Tree Rollout for Effective Multi-Turn RL

2026-07-17

AI Summary

Traditional multi-turn reinforcement learning wastes substantial sampling budgets on uninformative failed trajectories due to uniform sampling and insufficient exploration of high-potential intermediate states, especially in long-horizon tasks. This work proposes Process-feedback-guided Adaptive Tree-based Trajectory Rollout (PATR), a novel framework that, for the first time, models multi-turn agent trajectories as a tree structure. PATR leverages task-specific process rewards to dynamically evaluate partial trajectories, enabling selective expansion from high-potential states, reuse of shared trajectory prefixes, and pruning of degenerate paths. Compatible with standard policy optimization methods, PATR achieves performance gains of 5.0 and 9.3 points on SWE-Bench and FrozenLake, respectively, significantly enhancing both sampling efficiency and overall performance in complex multi-turn tasks.

Abstract

Reinforcement learning (RL) has become a key approach for training LLM agents, yet popular methods such as GRPO/RLOO rely on multiple independently sampled complete trajectories for advantage estimation. In long-horizon agentic tasks, such a uniform rollout strategy can waste budget on uninformative dead-end attempts, while promising intermediate states do not receive sufficient exploration. The multi-turn structure of agentic trajectories, with interleaved actions and observations, naturally supports organizing a trajectory group as a tree, where each turn serves as a decision point for exploration. This perspective reframes effective exploration as the problem of deciding where to branch. We propose Process-Scorer Guided Adaptive Tree Rollout (PATR), a quality-aware rollout framework for multi-turn agent RL. PATR uses task-appropriate process feedback to score partial trajectories, selectively branches from promising states, reuses shared prefixes, and conservatively stops degenerate paths to reduce wasted sampling. The resulting rollout groups remain compatible with standard policy optimization while providing more efficient exploration under the same training budget. We evaluate PATR on FrozenLake and the challenging SWE-Bench, which is largely unexplored by prior tree-rollout agent RL methods. Experiments show that PATR improves performance by up to +5.0 points on SWE-Bench and +9.3 points on FrozenLake, highlighting process-guided tree rollouts as an effective strategy for scalable multi-turn RL.

Problem

• multi-turn RL
• trajectory exploration
• sample efficiency
• long-horizon tasks
• dead-end trajectories

Innovation

• Tree Rollout
• Process Reward
• Multi-turn RL
• Adaptive Exploration
• Trajectory Scoring