EvoBrowseComp: Benchmarking Search Agents on Evolving Knowledge

2026-06-11

AI Summary

Existing benchmarks for evaluating search agents rely on static knowledge, rendering them susceptible to test-set contamination and parametric memory effects, thereby failing to accurately assess genuine retrieval and reasoning capabilities. To address this limitation, this work proposes the first dynamic evaluation paradigm that supports automatic updates, resists contamination, and presents high difficulty. The framework employs a three-agent collaborative architecture—comprising question-answer synthesis, information filtering, and high-level guidance—to generate high-quality, uncontaminated, complex bilingual (Chinese–English) question-answer pairs from real-time web sources. Logical graphs are formally integrated to block shortcut learning through memorization. Built upon this approach, the EvoBrowseComp benchmark comprises 800 challenging questions and demonstrates high difficulty and strong generalization in experiments, effectively distinguishing models’ authentic web-browsing reasoning abilities from mere reliance on parametric memory.

Abstract

Search Agents -- large language models augmented with search tools -- have intensified the need for future-proof evaluation benchmarks. Existing benchmarks such as BrowseComp rely on static knowledge, making them vulnerable to test-set contamination and parametric memorization. Consequently, models can achieve high scores through fact recall rather than genuine retrieval, obscuring true browsing competence via reasoning shortcuts. In this paper, we introduce EvoBrowseComp, an evolving benchmark of 400 English and 400 Chinese contamination-free complex questions synthesized via live-web traversal. To collect these questions, we design a three-agent collaborative framework: (1) a QA synthesis agent that retrieves fresh knowledge from the live web to synthesize QA pairs; (2) an information filtering agent that filters retrieved knowledge in terms of credibility and popularity to block parametric shortcuts; and (3) a high-level guidance agent that formalizes questions into reasoning graphs to reduce logical redundancy and shortcuts in synthesized QA pairs. Because the framework supports fully automated synthesis, EvoBrowseComp can be regularly updated to prevent data contamination and maintain temporal freshness. Extensive experiments confirm its great difficulty, requiring broad horizontal search. It establishes a scalable paradigm for auto-updatable, high-difficulty benchmarking that keeps pace with both evolving world knowledge and advancing agent capabilities.

Problem

• search agents
• evaluation benchmarks
• test-set contamination
• parametric memorization
• browsing competence

Innovation

• evolving benchmark
• search agents
• live-web traversal
• three-agent framework
• auto-updatable evaluation