From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents

2026-07-30

AI Summary

Current self-distillation approaches for large language model agents rely on action scores that lack environmental validation, hindering the internalization of reliable skills. This work proposes a self-distillation framework grounded in outcome verification and trajectory contrast. By dynamically invoking the teacher policy on student failure trajectories, the method retains only environment-validated successful continuations for localized contrastive learning. It further incorporates skill-conditioned teacher invocation, state-aligned divergence point detection, and a suffix distillation mechanism, while efficiently organizing failure trajectories using prefix trees. Evaluated on ALFWorld and WebShop, the approach improves success rates by 29.7 and 5.4 percentage points, respectively, with less than a 3% increase in privileged interactions.

Abstract

Recent work on LLM agents is shifting from external capability elicitation to capability internalization, enabling agents to retain useful skills without retrieval at inference time. On-policy self-distillation (OPSD) offers a promising direction, but many existing methods typically supervise students by scoring actions along student-generated trajectories. Such supervision has two limitations: teacher preferences are not validated by environment outcomes, and action-level scores underuse information from student rollouts, teacher rollouts, and their behavioral relationship. We therefore advocate outcome-verified teacher supervision and comparative learning over teacher-student trajectories. Based on this view, we propose Outcome-Verified Comparative Self-Distillation (OVCSD). OVCSD organizes failed student rollouts into a prefix tree, adaptively invokes a skill-conditioned teacher from student-reached states, and retains only outcome-verified successful continuations. It then applies localized comparative learning at the first state-aligned divergence and distills the post-divergence teacher suffix to transfer completion behavior. Experiments on ALFWorld and WebShop across three model scales show that OVCSD consistently outperforms skill-free RL and existing self-distillation baselines, achieving up to 29.7 and 5.4 absolute success-rate gains over the strongest baselines on ALFWorld and WebShop, respectively, while adding less than 3% privileged interaction during training.

Problem

• self-distillation
• LLM agents
• outcome verification
• comparative learning
• trajectory alignment

Innovation

• outcome-verified supervision
• comparative self-distillation
• trajectory divergence
• skill internalization
• prefix tree organization