Optimizing What Policies Learn From: Recoverability-aware Rollout Intervention Learning

2026-08-05

AI Summary

Existing post-training methods in reinforcement learning lack dynamic, fine-grained control over the timing and location of interventions during rollout generation, often resulting in redundant or inefficient learning signals. This work proposes RAIL, a novel framework that introduces, for the first time, a recoverability-aware mechanism to model intervention decisions as an online contextual bandit problem. RAIL employs a shadow-to-live pipeline to collect intervention trajectories and leverages a critic-free population-based reinforcement learning architecture to train a recoverability controller that continuously adapts alongside the evolving policy. This controller autonomously determines when and where to intervene, optimizing rollout quality under limited computational budgets. Empirical results demonstrate that RAIL significantly enhances post-training efficiency across diverse scenarios, yielding high-quality rollouts with greater informational content and reduced redundancy.

Abstract

Critic-free group-based reinforcement learning has become a scalable approach for post-training large language models. However, most existing methods allocate the same number of rollouts to every task and trajectory state, even though some rollouts provide much more useful learning signals than others. Recent work has started to treat rollout generation as an adaptive decision, but two important limitations remain. First, intervention strategies are often based on fixed heuristics and therefore cannot adjust as the policy changes during training. Second, these methods usually decide only how many rollouts to generate, without explicitly controlling where and how to intervene. To address these limitations, we propose Recoverability-Aware Intervention Learning (RAIL), a training-time framework that learns how to generate rollouts based on the improvement produced by each intervention. RAIL models intervention selection as an online contextual-bandit problem and trains a recoverability controller using intervention traces collected through a shadow-to-live procedure. This allows the controller to keep learning while the underlying policy evolves. We evaluate RAIL in terms of effectiveness, adaptivity, expressiveness, and efficiency. Across multiple settings, RAIL consistently improves performance under limited rollout budgets. These results show that recoverability-aware intervention provides a principled way to generate more informative and less redundant rollouts, leading to stronger learning signals during post-training.

Problem

• rollout intervention
• recoverability-aware learning
• adaptive decision-making
• post-training
• reinforcement learning

Innovation

• Recoverability-Aware Intervention Learning
• contextual bandit
• rollout optimization
• adaptive intervention
• post-training