Why Does Feedback-Augmented Self-Distillation Fail to Improve Retrieval-Interleaved Search Agents?

2026-07-20

AI Summary

In retrieval-augmented search agents, feedback-enhanced self-distillation fails due to the model’s reliance on fixed output templates, leading to decoding collapse and generation of trajectories unrelated to the input. This work reveals that this phenomenon is overlooked by existing evaluation metrics and, for the first time, attributes the inconsistency in supervision signals to both the model architecture and prompting strategy. To address this, the authors propose integrating an exponential moving average (EMA) teacher model to stabilize the distillation signal, combined with KL-divergence-based supervision and trajectory analysis for optimization. Although the EMA teacher requires a warm-up phase, the approach ultimately yields significant performance gains on complex search tasks, demonstrating its effectiveness.

Abstract

On-policy self-distillation (OPSD) offers a promising approach for training large language models without relying on a separate teacher model. However, its effectiveness on complex agentic tasks remains largely unexplored. In this work, we instantiate Feedback-Augmented Self-Distillation (FA-SD), a self-distillation algorithm for agentic search that leverages successful demonstrations as privileged information. We identify that models can rely on recurring reasoning-and-search output templates, producing trajectories that appear diverse but are largely agnostic to the input question, making the KL-based self-distillation signal uninformative. We term this phenomenon decoding collapse, a failure mode that can be missed by existing evaluation metrics. To understand its underlying cause, we show that although the self-teacher achieves stronger performance, learning remains inherently unstable due to inconsistent supervision signals. We further decompose this inconsistency into model inconsistency and prompt inconsistency, and show that the latter can significantly degrade the quality of the supervision signal, limiting the effectiveness of self-teacher learning. To mitigate this inconsistency, we introduce an exponential moving average (EMA) teacher to stabilize the self-teacher and provide more consistent supervision signals. Although the EMA teacher requires a warm-up phase during which performance may temporarily regress, it ultimately improves model performance by providing more stable supervision.

Problem

• self-distillation
• retrieval-interleaved search agents
• decoding collapse
• supervision inconsistency
• on-policy learning

Innovation

• self-distillation
• decoding collapse
• retrieval-interleaved search
• exponential moving average teacher
• supervision inconsistency