EviBack: Search-Agent Reinforcement Learning via Evidence-Constrained Teacher Backoff

2026-07-26

AI Summary

This work addresses the challenge in reinforcement learning where trajectories with all-zero rewards lack effective supervision, hindering the optimization of search behavior. To overcome this, the authors propose EviBack, a method featuring an evidence-constrained teacher fallback mechanism that provides auxiliary supervision for unproductive trajectories while preserving verifiable rewards. The approach decouples evidence evaluation from answer generation and introduces a novel two-stage automated teacher generation pipeline, which prevents reference answers from masking insufficient evidence. Integrated with a GPT-5.5–driven automatic prompt engineering (APE) framework, the method employs task decomposition, ablation analysis, and gating mechanisms to construct a dual-task teacher model. Experiments demonstrate consistent and significant F1 improvements across seven open-domain question answering benchmarks and three Qwen3 model scales, alongside reduced search iterations, fewer repeated queries, and less forced termination.

Abstract

Reinforcement learning enables Agentic RAG systems to learn multi-turn search from verifiable outcome rewards, but all- zero rollout groups provide no comparative signal and may hide useful search behavior. We present EviBack, an evidence- constrained Teacher backoff that supplies auxiliary super- vision to such groups while preserving verifiable Actor re- wards. It separates evidence assessment from answer refine- ment, preventing reference answers from overriding evidence- insufficiency judgments. A fully automated, end-to-end GPT- 5.5-assisted APE pipeline starts from a manually authored single-prompt dual-task Teacher, automatically partitions and labels rollout data, and performs ablation, task decomposition, evaluation, and selection to produce a gated two-stage Teacher. Compared with the manual design, the resulting Teacher im- proves downstream F1 and valid-answer rate while reduc- ing search, duplicate queries, and forced termination. Across seven open-domain QA benchmarks and three Qwen3 scales, EviBack improves F1 over Search-R1 and raises both single- and multi-hop macro F1. We guarantee that the code will be made publicly available at a later stage.

Problem

• Reinforcement Learning
• Agentic RAG
• Rollout Groups
• Evidence Assessment
• Zero-Reward Signal

Innovation

• Evidence-Constrained Teacher Backoff
• Agentic RAG
• Reinforcement Learning
• Automatic Prompt Engineering
• Multi-turn Search