Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluation

2026-06-03

AI Summary

This study systematically defines and quantifies "Search-Time Contamination" (STC)—a previously unaddressed issue in evaluating deep reasoning agents on public benchmarks, where real-time web search inadvertently inflates performance. The authors categorize STC into three types: metadata leakage, context exposure, and direct answer retrieval. Through formal modeling of contamination mechanisms, analysis of search trajectories, development of detection algorithms, and implementation of isolated benchmark evaluations, they demonstrate that STC is pervasive across six widely used benchmarks, artificially inflating agent performance by up to 4%. The findings reveal that current evaluation protocols may substantially overestimate agents’ true reasoning capabilities. To address this, the work proposes contamination-aware evaluation guidelines to foster more reliable and trustworthy assessment practices for intelligent agents.

Abstract

Public benchmarks enable fair and reproducible evaluation of LLM reasoning, but they become fragile for deep research agents that actively search the web during inference. Such agents may retrieve public benchmark metadata, question context, or even ground-truth answers via web search. This gives rise to Search-Time Contamination (STC), where external retrieval bypasses intended reasoning and inflates measured performance. We systematically study STC in deep research agent evaluation. We define three contamination types with increasing severity, namely Benchmark Metadata Leakage, Question-Context Leakage, and Explicit Answer Leakage, and develop detection algorithms to identify them and quantify their impact on agent performance. Evaluating modern deep research agents on six public benchmarks, we find that STC is widespread and can inflate performance by up to 4%. Our findings show that existing evaluations may overestimate true reasoning ability. We therefore advocate contamination-aware practices, including isolated sandboxes, transparent search trajectories, and controlled benchmark access.

Problem

• Search-Time Contamination
• deep research agents
• public benchmarks
• performance inflation
• web search

Innovation

• Search-Time Contamination
• Deep Research Agents
• Benchmark Evaluation
• Performance Inflation
• Web Search Leakage