The Role of Feedback Alignment in Self-Distillation

2026-06-09

AI Summary

The design mechanism of feedback context in self-distillation remains unclear. This work proposes a step-aligned critique feedback mechanism that provides structured guidance precisely at points of reasoning errors while avoiding interference with correct behaviors, thereby enhancing learning efficiency. Within a self-distillation framework, the study systematically compares three feedback forms: binary rewards (GRPO), reference solutions, and step-aligned critiques, complemented by per-token advantage analysis to evaluate their effectiveness. Experimental results demonstrate that step-aligned critique improves performance by 16.11 points over GRPO and by 5.27 points over reference solutions on the Avg@12 metric, confirming the critical importance of aligning feedback with the structural trajectory of reasoning.

Abstract

Conditioning a language model on additional context, such as feedback on a previous attempt, typically improves its response. Self-distillation trains the model to retain this improvement when the context is not present. The method works by matching the model's output distribution under two settings: a student that sees only the question, and a self-teacher that also sees the context. What the model learns therefore depends on what context the self-teacher receives, yet the design of this context remains largely unexplored. We study context design for self-distillation by training a solver on feedback from a frozen critic. We compare three conditions: (i) a binary reward (GRPO), (ii) the reference solution, and (iii) a step-by-step critique aligned to the solver's reasoning trace. Step-aligned critique yields the largest gains, outperforming GRPO by 16.11 points and reference-solution-conditioned self-distillation by 5.27 points (Avg@12). Per-token advantage analysis reveals why: step-aligned feedback targets only the tokens where reasoning fails, leaving correct behavior intact. Conditioning on the reference solution, by contrast, pressures the model to change its behavior at every token (even correct steps) because an alternative derivation inevitably differs in phrasing and approach. This suggests that structural alignment between feedback and the solver's reasoning is a key driver of self-distillation effectiveness.

Problem

• self-distillation
• feedback alignment
• context design
• language models
• reasoning trace

Innovation

• feedback alignment
• self-distillation
• step-aligned critique
• reasoning trace
• language model conditioning