Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents

2026-07-27

AI Summary

This work addresses a critical gap in evaluating code agents: while existing benchmarks emphasize final patch correctness, they largely overlook the foundational capability of repository file retrieval. To bridge this gap, we propose the first file-level retrieval benchmark tailored for code agents, introducing a novel relevance criterion grounded in the agent’s immediate next-step needs. The benchmark encompasses four positive task types and one selective task, augmented with natural samples lacking gold labels and counterfactual erroneous repositories for robust evaluation. Evaluating lexical retrieval, RepoMap, and Qwen3-Embedding methods—alongside real agent trajectories—on 427 samples reveals that Qwen3-Embedding-4B achieves the highest MRR, while RepoMap demonstrates superior context efficiency under an 8K-token budget. Initial retrieval substantially improves subsequent file-level F1 scores, yet overall performance remains far from ideal.

Abstract

Modern coding agents are usually evaluated by whether they eventually produce a correct patch, but patch generation depends on an earlier context-acquisition stage: finding the repository files needed for the task. We introduce Agent Retrieval Bench, a file-level benchmark for this upstream retrieval problem. Samples are built from real coding-workflow signals and evaluated against frozen base-commit repositories, with relevance defined by what an agent needs next rather than direct query-file semantic similarity. The benchmark covers four positive-retrieval tasks: code2test, comment2context, trace2code, and edit2ripple; a fifth subset evaluates selective retrieval using natural evidence-backed no-gold cases and counterfactual wrong-repository controls. Agent Retrieval Bench contains 427 samples across 25 repositories: 345 positive examples, 50 natural no-gold examples, and 32 counterfactual controls. The corpus includes 308 base-commit snapshots, 392,000 files, and 7.9 million chunks. We evaluate lexical retrieval, RepoMap, open-source embeddings, selective abstention, and logged agent context selection. No single retrieval family dominates: Qwen3-Embedding-4B has the best sample-weighted MRR on positive samples, Qwen3-Embedding-8B the best Recall@20, and RepoMap the best budgeted context yield at 8K tokens, with task-level winners differing substantially. Selective thresholds calibrated with counterfactual controls do not improve selective success on natural no-gold cases, revealing a calibration gap. Logged trajectories also miss every gold file on 27-35 percent of samples. A controlled seed-intervention pilot finds that retrieval-derived initial context yields higher file F1 with less post-seed exploration than random non-gold context, while oracle gold context shows substantial remaining headroom.

Problem

• code retrieval
• coding agents
• repository context
• retrieval benchmark
• context acquisition

Innovation

• context retrieval
• coding agents
• benchmark
• selective retrieval
• repository-level evaluation