From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolution

2026-08-03

AI Summary

This work addresses the limitations of existing deep research benchmarks, which predominantly rely on manual construction and thus struggle to support fully automated, verifiable, and traceable evaluation. To overcome this, the authors propose an automated framework based on an iterative Explorer-Formalizer-Challenger pipeline, enabling the first large-scale benchmark comprising 500 deep research tasks across 31 topics and 10 categories. Tasks are structured as directed acyclic graphs (DAGs) that formalize procedural steps and verification checkpoints, integrated with a fact-based, point-wise scoring mechanism. This design facilitates the co-evolution of queries, task graphs, and evaluation criteria. Empirical results demonstrate that the benchmark effectively discriminates between models and query types, yielding fine-grained, stable assessments that exhibit strong alignment with human judgment.

Abstract

Deep research benchmarks require expert-level tasks and reliable evaluation grounded in task-specific knowledge. Existing benchmarks rely heavily on expert authoring or pre-existing human-authored materials, while fully automatic construction struggles to ensure consistent and traceable verification. To address this gap, we introduce a verifiable benchmark of 500 deep research tasks spanning 31 topics and 10 major categories, with three query forms designed to probe complementary capabilities required for deep research. The benchmark is constructed automatically using an iterative Explorer-Formalizer-Challenger pipeline that progressively transforms simple questions into deep research tasks. Each task is represented as a directed acyclic graph (DAG) of atomic steps and associated checkpoints, enabling the query, DAG, and rubrics to evolve together in a controlled manner. Experiments demonstrate that the benchmark clearly discriminates among models and query types, while its fact-grounded pointwise rubrics enable fine-grained, human-aligned, and stable evaluation. Our data, implementation, and results are publicly available.

Problem

• deep research benchmark
• verifiable evaluation
• automatic construction
• expert-level tasks
• traceable verification

Innovation

• iterative task evolution
• verifiable benchmark
• deep research tasks
• directed acyclic graph (DAG)
• fact-grounded evaluation