Search-GRT: Guided Retrieval Training of Search Agents to Optimize for Complex Question Answering

2026-08-01

AI Summary

This work addresses the challenges of cascading failures due to early retrieval errors and sparse rewards in reinforcement learning for multi-hop question answering. The authors propose Guided Retrieval Training (GRT), a novel approach that leverages ground-truth answers during reinforcement learning to constrain the retrieval space and focus attention on relevant documents, thereby strengthening the learning signal. By introducing a retrieval guidance mechanism grounded in factual information, GRT effectively mitigates reward sparsity and significantly enhances both sub-query generation and answer synthesis capabilities. Experimental results demonstrate that GRT outperforms state-of-the-art methods—such as Search-R1—by over 40% across multiple multi-hop QA benchmarks, achieving superior performance with substantially fewer training steps.

Abstract

The effective use of search engines by large language models (LLMs) remains a significant challenge, particularly in complex, multi-hop question-answering (MHQA) tasks. These tasks require the model to decompose questions into subqueries, retrieve relevant information, and synthesize answers from multiple sources, often leading to cascading errors due to poor retrieval in early stages. Reinforcement learning (RL) has shown promise in improving LLMs' search capabilities, but it often suffers from sparse rewards during training, hindering the model's ability to learn effectively. To address these challenges, we introduce Guided Retrieval Training (GRT), a novel method that improves the performance of a search agent by restricting the retrieval process during RL training using ground truth information. By focusing on a curated set of relevant documents, GRT provides the model with a stronger learning signal, mitigating the problem of sparse rewards and improving its ability to generate accurate subqueries and synthesize correct answers. Our experimental results demonstrate that GRT achieves consistent performance improvements over existing methods, such as Search-R1, across a wide range of question-answering (QA) tasks. Notably, GRT excels in MHQA tasks, achieving over 40% improvements in performance. Additionally, GRT enhances training efficiency by achieving better QA performance with fewer training steps.

Problem

• multi-hop question answering
• search agents
• reinforcement learning
• sparse rewards
• retrieval

Innovation

• Guided Retrieval Training
• Reinforcement Learning
• Multi-hop Question Answering
• Sparse Rewards
• Search Agents