HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learning

2026-08-02

AI Summary

This work addresses the limitation of conventional search-augmented language model agents that rely solely on binary exact-match rewards, thereby ignoring fine-grained feedback from failed trajectories. The authors propose HindSearch, a method that introduces a frozen critic within the GRPO framework to generate trajectory-level post-hoc critiques using ground-truth answers. These critiques serve as auxiliary distillation signals to guide the student policy in optimizing search actions. HindSearch is the first to incorporate a ground-truth–based, trajectory-level post-hoc self-distillation mechanism into search-augmented reinforcement learning, substantially improving both learning efficiency and performance. Evaluated across seven standard benchmarks, the method achieves an average exact-match accuracy of 39.4%, significantly outperforming existing search-RL baselines. Ablation studies confirm the critical role of post-hoc critique, as removing the critic’s access to ground-truth answers leads to a marked performance drop.

Abstract

Search-augmented LM agents are typically trained with a binary exact-match reward, which throws away most of what a failed trajectory tells us about why it failed. We introduce HindSearch, a hindsight self-distillation procedure for GRPO: after each rollout, a frozen judge writes a short critique of every failed trajectory using the gold answer, and the critique supplies an auxiliary on-policy distillation signal on the student's search actions. On the standard seven-benchmark suite with Qwen2.5-3B-Instruct, HindSearch reaches 39.4% average EM, outperforming prior search-RL baselines. Removing the judge's access to the gold answer erases most of the gain, isolating hindsight as the source of the improvement.

Problem

• search-augmented reinforcement learning
• hindsight critique
• trajectory-level feedback
• reward sparsity
• failed trajectories

Innovation

• hindsight critique
• search-augmented reinforcement learning
• self-distillation
• trajectory-level feedback
• GRPO