Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents

2026-08-03

AI Summary

This study addresses critical challenges in long-horizon search agents, particularly the unclear relationship between search effort and answer quality, as well as failures in evidence retrieval and utilization. It introduces a systematic framework to distinguish and quantify two distinct gaps—“retrieval gap” and “utilization gap”—leveraging human-annotated document-level relevance, a fixed retrieval model, multi-agent comparisons, and an open-web search API for trajectory-level diagnosis. The findings reveal that answer accuracy is primarily determined by the quality of retrieved evidence rather than the number of search queries. Moreover, high-performing agents tend to minimize redundant queries, with crucial evidence typically appearing in early search stages. These insights provide both theoretical grounding and practical guidance for improving deep research systems.

Abstract

Deep search agents answer difficult information-seeking questions by iteratively issuing search queries to gather supporting evidence, but it remains unclear whether and how greater search effort leads to better answers. We study these questions through a trajectory-level diagnosis of long-horizon search agents. Using human-annotated document-level relevance judgments, we evaluate the evidence retrieved at each search step and separate two stages of agent behavior: what evidence an agent retrieves and how effectively it uses that evidence. This distinction further allows us to decompose failures into retrieval gaps, where the necessary evidence is never found, and utilization gaps, where relevant evidence is retrieved but not used correctly. With the retrieval model and evaluation harness held fixed, we compare six agents on BrowseComp-Plus and further validate our findings on BrowseComp with an open-web search API. Across settings, we find that search effort and answer quality are only weakly aligned. Answer accuracy is better correlated with the quality of retrieved evidence, especially cumulative retrieval recall, than with the number of searches or the amount of context consumed. Useful evidence often appears early in the trajectory, yet agents tend to continue searching, producing a long tail of low-yield retrieval steps. At the query level, exploratory reformulations remain useful, but the best-performing agents issue far fewer redundant queries. Overall, by systematically characterizing the search behavior and failure modes of long-horizon search agents, this work points to practical directions for building better deep research systems, including stronger query formulation, more effective evidence selection and context management, and stopping criteria based on whether sufficient supporting evidence has been retrieved.

Problem

• search behavior
• failure modes
• long-horizon search agents
• evidence retrieval
• answer quality

Innovation

• long-horizon search agents
• trajectory-level diagnosis
• retrieval-utilization gap
• cumulative retrieval recall
• query reformulation