TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoning

2026-08-04

AI Summary

This work addresses the limitations of existing reinforcement learning methods in tool-augmented reasoning, which rely on trajectory-level supervision and struggle with fine-grained credit assignment in long-horizon tasks while underutilizing actually visited states. To overcome these challenges, the authors propose TurnSight, a turn-level hindsight self-distillation framework that introduces, for the first time, execution-conditioned turn-level hindsight supervision. TurnSight constructs multi-view hindsight representations and adaptively modulates the advantage function by leveraging cross-horizon directional consistency to filter reliable signals. It further incorporates a sibling-trajectory normalization mechanism to enable more precise credit assignment and policy optimization. Experiments across three benchmarks demonstrate that TurnSight substantially improves performance, confirming its effectiveness and robustness in complex tool-interaction tasks.

Abstract

Tool-Integrated Reasoning (TIR) enables LLMs to solve complex tasks through iterative tool interactions. However, existing reinforcement learning methods often rely on trajectory-level supervision, limiting fine-grained credit assignment in long-horizon TIR scenarios. On-policy self-distillation offers denser signals through teacher branches with privileged context, but existing approaches typically derive such context from ground-truth answers or retrieved skills, which may not reflect the states actually visited by the agent. Moreover, token-level supervision fails to capture the turn-level structure of tool interactions. To address this, we propose TurnSight, a turn-level hindsight self-distillation framework that derives supervision directly from execution-conditioned hindsight. It then constructs multiple hindsight views with different lookahead horizons and selects reliable supervision through cross-horizon directional agreement. Finally, the selected hindsight signal is normalized across sibling rollouts and used to adaptively modulate RL advantages while preserving their original optimization direction. Extensive experiments on three benchmarks demonstrate the effectiveness of TurnSight. Our codes are available at https://github.com/quchangle1/TurnSight.

Problem

• Tool-Integrated Reasoning
• credit assignment
• self-distillation
• turn-level supervision
• hindsight

Innovation

• hindsight self-distillation
• turn-level supervision
• tool-integrated reasoning
• reinforcement learning
• credit assignment