Group-Reflective Self-Distillation for Agentic Reinforcement Learning

2026-07-30

AI Summary

In reinforcement learning with verifiable rewards, terminal rewards provide only trajectory-level supervision, making fine-grained credit assignment challenging. To address this, this work proposes an intra-group reflective self-distillation mechanism that leverages verified trajectories generated by the policy itself. By contrasting self-reflection outcomes between successful and failed trajectories within the same group—and integrating stop-gradient snapshots with advantage-function modulation—the method constructs a capability-aligned, cross-trajectory guidance signal. Notably, it requires no external skills or strong supervision. Extensive experiments demonstrate that the approach significantly outperforms existing baselines across diverse environments and model scales, while exhibiting superior generalization capabilities.

Abstract

Reinforcement learning with verifiable rewards (RLVR) is effective for training large language model agents. However, terminal rewards provide only coarse trajectory-level supervision, leaving successful behaviors, recurring mistakes, and incidental choices entangled in the same outcome signal. Existing agentic self-distillation methods enrich sparse supervision with natural-language skills, but skills retrieved externally or extracted from a single trajectory by stronger models may mismatch current experience, exceed the policy's capability, or remain path-specific. We propose Group-Reflective Self-Distillation (GRSD), which derives capability-aligned and outcome-discriminative guidance from the policy's own verified rollouts. For each prompt, the policy reflects on each verified trajectory in an on-policy group, and a stop-gradient snapshot contrasts the resulting reflections from successful and failed rollouts to construct group-level privileged guidance. Conditioned on this guidance, a self-teacher refines turn-level credit assignment by modulating outcome-based advantages while preserving the verifier-determined learning direction. Experiments across multiple agentic environments and model scales demonstrate that GRSD consistently outperforms competitive baselines and generalizes more effectively to unseen tasks.

Problem

• reinforcement learning
• self-distillation
• trajectory-level supervision
• capability alignment
• outcome discrimination

Innovation

• Group-Reflective Self-Distillation
• agentic reinforcement learning
• self-distillation
• credit assignment
• capability-aligned guidance