DASH-OPD: Discrepancy-Aware Switching with Hysteresis for On-Policy Distillation

2026-07-31

AI Summary

This work addresses the challenge in multi-turn agent settings where early errors by a student model can cause trajectory deviations outside the teacher’s familiar domain, and existing curriculum learning approaches struggle to dynamically determine when teacher intervention is needed. To this end, the paper proposes an adaptive bidirectional actor-switching mechanism tailored for policy distillation. The method quantifies discrepancies between actors using the average log-probability ratio of action logits, accumulates evidence of drift and recovery across multiple episodes, and employs hysteresis thresholds to suppress frequent switching triggered by transient fluctuations. By uniquely integrating cumulative discrepancy signals with hysteresis-based control, the approach enables dynamic perception of teacher support needs and stable responses, significantly outperforming existing baselines in the ALFWorld environment while achieving superior efficiency in both training and deployment.

Abstract

On-policy distillation (OPD) trains student models on their own rollouts to reduce exposure bias. However, in multi-turn agent scenarios, early student errors can lead a trajectory away from the teacher's familiar domain. Existing curriculum learning methods regulate how much teacher support is used according to training progress, but cannot determine when it is needed. In light of this, we propose DASH-OPD, Discrepancy-Aware Switching with Hysteresis for OPD, a new agentic OPD method that can switch executors adaptively and bidirectionally. On each turn, DASH-OPD calculates a mean log-probability ratio between the two executors over action tokens as their discrepancy. Student-to-teacher ratios on student turns form drift signals, while teacher-to-student ratios on teacher turns form recovery signals. These signals are normalized and accumulated over multiple turns into drift and recovery evidence. DASH-OPD switches executors when the evidence exceeds its corresponding switching threshold. This multi turn accumulation makes the switching hysteretic, preventing high-frequency switches caused by transient fluctuations. On ALFWorld, DASH-OPD outperforms all the baselines and demonstrates superior training and deployment efficiency. This paper is a work in progress. Code, training logs, and model checkpoints will be released later.

Problem

• On-policy Distillation
• Exposure Bias
• Multi-turn Agents
• Curriculum Learning
• Executor Switching

Innovation

• On-Policy Distillation
• Discrepancy-Aware Switching
• Hysteresis
• Multi-turn Agents
• Adaptive Executor Switching