Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning

2026-08-07

AI Summary

In reinforcement learning, sparse rewards hinder effective credit assignment across decision points in a trajectory, limiting learning efficiency. This work proposes the TRIAL framework, which introduces a novel trajectory-relative hindsight distillation mechanism: for each decision step, it constructs an outcome-aware perspective and computes the signed difference between token-level log-probabilities under standard and hindsight conditions as a supervision signal. This signal is jointly normalized over the entire trajectory to dynamically redistribute supervision strength while preserving the average multiplier. By integrating hindsight experience replay with dense supervision distillation, TRIAL consistently outperforms GRPO across eight model-environment configurations on WebShop and ALFWorld. Notably, with Qwen3-1.7B on WebShop, it improves success rate from 56.4% to 75.2% and task score from 78.7% to 85.7%.

Abstract

Recent agentic reinforcement learning methods use hindsight to complement sparse outcome rewards. However, a completed rollout can yield many such signals, leaving their appropriate allocation across turns unclear. We introduce TRIAL, a trajectory-relative hindsight distillation framework with a unified turn-aligned scoring protocol. For each decision turn, TRIAL extracts an outcome view of that decision's realized consequence and evaluates the same response under ordinary and hindsight-conditioned contexts. The signed log-probability gap determines the direction and local strength of token-level supervision, while turn-level magnitudes are normalized jointly over the realized trajectory. The resulting allocation multipliers have an eligible-token-weighted mean of one, redistributing dense supervision across turns while fixing its average multiplier. Experiments on WebShop and ALFWorld with different backbones show that TRIAL outperforms GRPO across all eight combinations of backbone, environment, and evaluation metric, while achieving the best or tied-best performance among six methods on six of them. On WebShop with Qwen3-1.7B, TRIAL improves the success rate from 56.4% to 75.2% and the task score from 78.7% to 85.7%. Controlled ablations further show that trajectory-relative turn allocation provides substantial gains beyond those of dense hindsight distillation alone.

Problem

• agentic reinforcement learning
• hindsight
• reward allocation
• trajectory
• sparse rewards

Innovation

• hindsight distillation
• trajectory-relative allocation
• turn-aligned scoring
• agentic reinforcement learning
• dense supervision redistribution