BiCAA: Bidirectional Credit Assignment for Search-Augmented Agent

2026-08-02

AI Summary

This work addresses the challenge in traditional reinforcement learning where sparse terminal rewards in multi-step search tasks hinder effective supervision of intermediate reasoning steps, leading to unstable training and redundant exploration. To overcome this limitation, the paper proposes BiCAA, a novel framework that introduces, for the first time, a bidirectional credit assignment mechanism. BiCAA integrates forward solvability gain with hindsight criticality scoring to deliver fine-grained, discriminative process rewards for each search step. This enables dense supervision of intermediate actions, substantially improving both search efficiency and policy stability. Experimental results on search-augmented question answering benchmarks demonstrate that BiCAA effectively reduces redundant behaviors, stabilizes policy optimization, and achieves competitive performance.

Abstract

Multi-step search is a fundamental capability for search agents, enabling them to iteratively acquire, refine, and integrate external evidence for complex reasoning QA. However, vanilla GRPO allocates rewards exclusively based on the model's final outputs, yielding outcome-only supervision with no supervisory signals for intermediate reasoning steps. Such sparse supervision easily causes training instability and redundant search behaviors on multi-step search tasks. To mitigate this limitation, we adopt process reward to deliver stepwise supervision signals. For this process reward, we propose two complementary criteria to judge each search step: whether the step yields new evidence to facilitate problem solving, and whether it forms an efficient, pivotal intermediate decision within the overall reasoning trajectory. Building on this insight, we propose BiCAA: a bidirectional credit assignment framework that delivers dense, distinguishing process rewards for search-augmented agents. BiCAA builds bidirectional process rewards by fusing two complementary signals: forward solvability gain and hindsight success criticality. The former quantifies step-wise improvements in answer plausibility, while the latter evaluates each step's necessity for final success via hindsight outcome-based criticality scoring. We modulate and aggregate the two signals and then fuse them with the outcome reward. Experiments on search-augmented QA benchmarks show that BiCAA stabilizes policy optimization, reduces redundant search behavior, and achieves competitive performance.

Problem

• credit assignment
• search-augmented agent
• multi-step search
• process reward
• reinforcement learning

Innovation

• Bidirectional Credit Assignment
• Process Reward
• Search-Augmented Agent
• Multi-step Reasoning
• Hindsight Criticality