EviSD: Evidence-Conditioned Self-Distillation for Search-Augmented Agents

2026-08-02

AI Summary

Existing outcome-based reinforcement learning struggles to quantify the specific contribution of individual actions to the final answer in multi-step search processes. To address this, this work proposes EviSD, a novel framework that introduces, for the first time, an evidence-conditioned self-distillation mechanism into retrieval-augmented agents. By leveraging instance-level supporting evidence and ground-truth answers as privileged information, EviSD enables a single model to act simultaneously as both student and teacher—operating in original and evidence-aligned contexts, respectively—and applies bounded corrections to generated actions. This approach delivers localized privileged guidance without requiring auxiliary objectives or modifications at inference time. Combined with GRPO advantage correction and action-aligned re-ranking, EviSD achieves state-of-the-art macro-averaged Exact Match across seven question-answering benchmarks and three backbone models, outperforming the strongest baseline by 1.3–2.3 points while adjusting only 6.7%–15.1% of response tokens.

Abstract

Outcome-based reinforcement learning enables search-augmented language agents to learn from verifiable final answers, but its trajectory-level credit cannot distinguish the contributions of individual actions in a multi-turn search process. We propose EviSD, an evidence-conditioned self-distillation framework that uses instance-level supporting evidence as privileged information for search actions and golden answers as complementary privilege for answer actions. During training, the student samples actions from the original context, while the same model re-scores them as a privileged teacher under an action-aligned context. EviSD converts the detached teacher--student gap into a bounded correction to the outcome-derived GRPO advantage and applies it only to generated action spans. This design localizes privileged guidance while preserving the update direction determined by the outcome reward, without an auxiliary distillation objective or any change at inference time. Across seven question-answering benchmarks and three backbones spanning model scales and generations, EviSD achieves the highest macro-average Exact Match in all evaluated settings, outperforming the strongest compared methods by 1.3--2.3 points while modulating only 6.7%--15.1% of response tokens. Code is available at https://github.com/JiananXie/EviSD.

Problem

• search-augmented agents
• credit assignment
• reinforcement learning
• multi-turn search
• trajectory-level credit

Innovation

• Evidence-Conditioned Self-Distillation
• Search-Augmented Agents
• Privileged Information
• GRPO Advantage Correction
• Trajectory-Level Credit Assignment