Harness-G: A Graph-Structured Harness for Search Agents

2026-07-30

AI Summary

This work addresses the retrieval aliasing problem in reinforcement learning–based retrieval agents, where diverse free-form queries yield highly overlapping evidence sets, thereby weakening action contrast signals. To mitigate this issue, the authors propose Harness-G, a graph-structured retrieval framework that reformulates query generation as a constrained action selection over a menu of evidence sentences or entities provided by the environment. The framework incorporates a Structured Non-myopic Credit assignment (SNC) mechanism, which freezes the answer scorer to compare each action against its alternatives and propagates downstream rewards back to early-stage decisions. By integrating graph-based action space modeling, state tracking, and verification mechanisms, Harness-G substantially outperforms the strongest existing baseline, Graph-R1, achieving average F1 improvements of 10.74 and 3.98 percentage points on six question-answering benchmarks with 1.5B and 3B parameter models, respectively.

Abstract

Reinforcement learning (RL) search agents commonly model retrieval as free-form natural-language query generation and optimize multi-turn interactions using final-answer rewards. Current studies mainly improve training with denser or more structured credit signals, but rarely examine whether retrieval is properly formulated at the policy-environment interface. We observe pronounced retrieval aliasing during Search-R1 training: rollouts for the same question continue to generate distinct query strings, yet their accumulated evidence sets increasingly overlap. We call this phenomenon retrieval-equivalence collapse; in this regime, trajectories approach utility equivalence with respect to retrieval decisions, leaving within-group returns with little effective retrieval contrast. To address this problem, we propose Harness-G, a graph-structured retrieval framework that redesigns this interface. It reformulates free-form query generation as finite action selection: the policy selects an evidence sentence or entity, or chooses to answer, while the environment constructs the menu, tracks retrieval state, and validates and executes each choice. This interface reduces linguistic aliasing and makes same-state alternatives directly comparable. Building on this interface, we introduce Structured Non-myopic Credit (SNC), which uses a frozen answer scorer to compare the selected action with its alternatives and assigns downstream gains to the earlier actions that enabled them. Across six QA benchmarks, Harness-G achieves the highest average F1 at both evaluated model scales, outperforming the strongest baseline, Graph-R1, by 10.74 points at 1.5B and 3.98 points at 3B.

Problem

• retrieval-equivalence collapse
• search agents
• retrieval aliasing
• policy-environment interface
• reinforcement learning

Innovation

• graph-structured retrieval
• retrieval-equivalence collapse
• finite action selection
• structured non-myopic credit
• policy-environment interface