Contextual Information Policy Optimization for Search Agents

2026-08-06

AI Summary

Existing search agents in multi-hop reasoning are often dominated by prior knowledge, treating retrieval merely as a post-hoc verification step, which leads to confirmation bias and underutilization of evidence. This work proposes the Contextual Information Policy Optimization (CIPO) framework, which, for the first time, explicitly aligns post-retrieval reasoning actions with retrieved evidence without requiring human annotations or additional reward models. CIPO leverages turn-level signals of evidence usage combined with a global reward based on answer correctness to provide dense supervision, enabling the agent to dynamically adjust its reasoning based on external evidence. Evaluated across seven domains and cross-domain benchmarks, the method significantly mitigates prior-driven behavior and achieves state-of-the-art performance.

Abstract

Search agents extend large language models beyond static parametric memory by enabling them to acquire and use ex ternal evidence during multi-step reasoning. For knowledge intensive tasks involving complex or evolving information, their reliability depends not only on retrieving relevant ev idence but also on using it to guide subsequent reasoning. However, existing methods primarily reward final-answer cor rectness or intermediate progress, without directly assessing whether post-retrieval actions are grounded in the retrieved evidence. This misalignment encourages prior-driven reason ing: agents form conclusions based on internal knowledge and use retrieval mainly to confirm them, resulting in confirma tion bias and inefficient evidenceuse.Toaddressthisissue, we propose Contextual Information Policy Optimization (CIPO), an evidence-oriented reinforcement learning framework that explicitly aligns policy optimization with external evidence use. CIPO assigns dense, turn-level credit to reasoning ac tions influenced by retrieved information, while combining this evidence-use signal with a global outcome reward to pre serveanswercorrectness.Withthismanner,CIPOdiscourages evidence-detached guesses and promotes reasoning trajecto ries in which retrieved facts can guide or revise subsequent reasoning. Importantly, CIPO requires neither human process annotations nor an additional reward model. Extensive exper iments on seven in-domain and out-of-domain benchmarks show that CIPO reduces the prevalence of prior-driven rea soning and achieves excellent performance on most tasks.

Problem

• search agents
• evidence use
• prior-driven reasoning
• confirmation bias
• reasoning alignment

Innovation

• Contextual Information Policy Optimization
• evidence-grounded reasoning
• reinforcement learning
• search agents
• confirmation bias mitigation