Problem
Research questions and friction points this paper is trying to address.
on-policy distillation
agentic tasks
trajectory deviation
teacher guidance
distribution gap
Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation
🤖 AI Summary
This work addresses the challenge in multi-turn agent tasks where student policies, due to accumulated behavioral drift, often diverge from regions where teacher guidance remains effective, thereby undermining the utility of conventional on-policy distillation. To overcome this limitation, the authors propose FutureBridge-OPD, a novel approach that injects brief teacher interventions at high-disagreement states and dynamically decides whether to retain such guidance based on a forward-looking validation criterion—specifically, whether the student’s subsequent trajectory increases the density of positive distillation signals. This method pioneers the integration of future-trajectory utility assessment into the distillation process, moving beyond paradigms that rely solely on current-state alignment. Experiments demonstrate that FutureBridge-OPD achieves average performance gains of 16.6 and 7.6 percentage points over standard OPD and TCOD, respectively, across ALFWorld, WebShop, and ScienceWorld, with consistent robustness across varying student model sizes and teacher configurations.
📝 Abstract
On-policy distillation (OPD) provides teacher supervision on states visited by the student, reducing the distribution gap between training and inference. However, in multi-turn agentic tasks, student deviations may accumulate over time, gradually moving the trajectory away from states where teacher guidance remains effective. Our quantitative analysis further shows that high-disagreement states offer promising opportunities for teacher guidance, but determining whether such guidance is beneficial requires examining its effect on subsequent student trajectories. We propose FutureBridge-OPD (FTB), which executes a short teacher bridge at a high disagreement state and uses the resulting student continuation to assess whether the bridge increases the density of positive distillation signals relative to the teacher. On ALFWorld, WebShop, and ScienceWorld, under the main Qwen3-32B teacher to Qwen3-1.7B student setting, FTB outperforms vanilla OPD and TCOD by an average of 16.6 and 7.6 points, respectively, and remains effective across student scales and teacher settings. Our code is publicly available at https://github.com/ChenChiShui/FutureBridge-OPD.
Innovation
Methods, ideas, or system contributions that make the work stand out.
On-Policy Distillation
Future Trajectory Validation
Teacher Guidance
Agentic Tasks
Trajectory Bridging
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
