PROGRESS: Coverage-guided RL to Train Search-augmented LLM Agent

2026-08-01

AI Summary

This work addresses the limitation of existing retrieval-augmented large language model agents, which rely solely on outcome-level rewards during reinforcement learning and thus struggle to effectively supervise query decomposition and search behaviors. To overcome this, the authors propose PROGRESS, a method that introduces teacher-guided coverage rewards within the R1 training framework. By leveraging a frozen teacher model to generate critical sub-queries, PROGRESS provides lightweight, explicit supervision over the agent’s search process without requiring dense step-level annotations. This approach innovatively integrates coverage-based rewards into reinforcement learning to efficiently guide search strategies. Experimental results demonstrate that PROGRESS significantly improves task performance, underscoring the critical role of explicitly supervising search behavior in enhancing agent reasoning capabilities.

Abstract

Existing search-augmented LLM agents are trained using Reinforcement Learning to boost its reasoning capabilities. However, these approaches primarily rely on outcome-level rewards, which provide little supervision over search behavior and overlook agent's ability to decompose complex queries properly. To mitigate this issue, we propose PROGRESS which utilizes teacher-guided coverage reward to explicitly shape decomposed query generation of the policy model. During training, frozen teacher models are used to decompose complex queries into essential search queries. These essential search queries are utilized to guide the search behavior of the policy model. Integrated into an R1-style training framework, our approach provides lightweight guidance over query decomposition decisions without dense process-level supervision. Experiments show that coverage-guided RL improves overall task performance, highlighting the importance of explicitly supervising search strategies in agentic LLMs.

Problem

• search-augmented LLM
• query decomposition
• reinforcement learning
• coverage reward
• agent reasoning

Innovation

• coverage-guided RL
• search-augmented LLM
• query decomposition
• teacher-guided reward
• reinforcement learning