Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation

2026-07-31

AI Summary

This work addresses the limitations of Online Policy Distillation (OPD), which incurs high computational overhead and struggles to adapt across models and training phases due to its reliance on fixed or absolute thresholds for controlling rollback length. To overcome this, the authors propose a learning-progress-aware adaptive rollback expansion strategy that dynamically identifies learning stagnation by monitoring relative changes in teacher–student signals and decides whether to extend the rollback horizon based on rollback utilization, thereby eliminating the need for preset step sizes or absolute consistency thresholds. Evaluated on two teacher–student model pairs, the proposed method achieves state-of-the-art performance, reducing training time by 49.1%–71.2% compared to OPD within 15K training steps while maintaining robustness across diverse hyperparameter configurations.

Abstract

On-policy distillation (OPD) provides dense teacher supervision along student-generated trajectories, but its online rollout process incurs substantial computational cost, particularly when a few long responses delay batch completion. Existing acceleration methods typically control rollout length using fixed budgets or absolute teacher--student agreement thresholds, which may not reflect learning progress across different models and training stages. We propose Adaptive FastOPD, a progress-aware strategy that expands the rollout horizon only when learning near the current boundary region has plateaued and the current horizon is sufficiently utilized. The former is determined from four teacher--student signals measured relative to their values upon entering each horizon, making expansion responsive to stage-specific progress rather than a predefined step interval or an absolute threshold on the raw agreement signals, while the latter prevents a small number of long responses from triggering increases in rollout cost. Across two teacher--student pairs, Adaptive FastOPD achieves the highest average performance while reducing training time by 49.1--71.2\% relative to OPD 15K, and remains robust across a range of hyperparameter settings.

Problem

• on-policy distillation
• rollout horizon
• computational cost
• learning progress
• teacher-student agreement

Innovation

• Adaptive FastOPD
• on-policy distillation
• progress-aware
• rollout horizon expansion
• teacher-student signals