Effective Reinforcement Learning for Agentic Search by Recycling Zero-Variance Queries During Training

2026-06-09

AI Summary

In traditional reinforcement learning, zero-variance query groups—those yielding uniformly correct or incorrect responses—are often statically discarded, leading to inefficient sampling. This work proposes a dynamic query recycling mechanism that, for the first time, reveals how zero-variance states evolve with policy updates and reintegrates them into a variable query pool to support a co-evolving training distribution. Building upon a GRPO-style algorithm, the approach combines this dynamic pool with a resampling strategy to substantially enhance sample utilization efficiency. Evaluated on seven multi-hop question answering benchmarks, a 1.7B-parameter model achieves an average Pass@1 accuracy of 66.0%, matching or surpassing existing methods that rely on larger models or supervised data.

Abstract

The use of GRPO-style algorithms has become the standard strategy for training LLM search agents under outcome-only rewards. With these algorithms, a query contributes to parameter updates only when its rollout group mixes successes and failures; all-correct (too-easy) and all-incorrect (too-hard) groups are zero-variance and waste rollout cost. Existing approaches treat zero-variance as a static property and either discard or pre-filter such groups. We hypothesize and empirically validate that queries flip between zero-variance and signal-bearing states as the policy evolves during training. Building on this intuition, we propose query recycling, which returns zero-variance groups to a mutable pool for future resampling, so that the effective training distribution co-evolves with the policy. With the proposed technique, a 1.7B parameter model trained on synthetic data can reach 66.0 average Pass@1 accross seven multi-hop QA benchmarks, matching or surpassing systems with up to 7B parameters trained on benchmark-derived supervision. Analysis of recycling patterns shows that recycled queries supply roughly three quarters of the effective batch by the end of training, with contributions split between recovery from policy improvement and policy drift.

Problem

• reinforcement learning
• zero-variance queries
• agentic search
• LLM training
• query recycling

Innovation

• query recycling
• zero-variance queries
• GRPO
• co-evolving training distribution
• agentic search