Where the Model Changes Its Mind: Hindsight-Divergence Localization for Efficient Reinforcement Learning with Verifiable Rewards

2026-09-29

AI Summary

This study addresses the high computational cost of independently sampling complete trajectories and the insufficient exploration of critical decision points in Reinforcement Learning with Verifiable Rewards (RLVR). To this end, we propose HDL, a method that innovatively identifies divergence points via posterior log-likelihood variations. By reusing trajectory prefixes and generating only suffixes, HDL enables efficient, focused exploration within critical decision spaces. Integrated with reinforcement learning and verifiable reward mechanisms, our approach substantially reduces computational overhead across mathematical reasoning, code generation, and agent-based tasks. Empirical results demonstrate that HDL decreases generated tokens by 2.5×, accelerates inference speed by 1.8×, and achieves performance improvements of up to 12.5 points compared to standard baselines.

Abstract

Group-relative methods for reinforcement learning with verifiable rewards (RLVR) learn from differences in rollout outcomes. Independently sampling complete trajectories is costly and does not explicitly explore the decision space at critical positions. Feedback on a completed trajectory can reveal which earlier choices the policy reconsiders, suggesting where to sample alternative continuations. We introduce Hindsight-Divergence Localization (HDL), which uses hindsight-induced changes in token log-likelihoods to select branch points. HDL generates a small number of complete root trajectories and fills each training group with continuations from the selected positions under the original task context. Each continuation reuses its root prefix and contributes policy updates only through its newly generated suffix, reducing generation cost while focusing additional exploration and learning on decisions after branching. Experiments with three models across math, code, and agent tasks show gains in both rollout efficiency and task performance. Compared with GRPO at matched group sizes and training steps, HDL yields up to a 2.5$\times$ reduction in generated tokens and a 1.8$\times$ speedup in rollout wall-clock time. Despite this reduced generation budget, HDL improves performance across all three domains, with gains of up to 12.5 points on agent tasks.

Problem

• Reinforcement Learning with Verifiable Rewards
• Rollout Efficiency
• Trajectory Sampling
• Decision Space Exploration

Innovation

• Hindsight-Divergence Localization
• Reinforcement Learning with Verifiable Rewards
• Branch Point Selection
• Rollout Efficiency
• Token Log-likelihood