Multi-Turn Evaluation of Deep Research Agents Under Process-Level Feedback

2026-06-08

AI Summary

This study addresses the limitation of existing evaluations of Deep Research Agents (DRAs), which focus solely on single-round outputs and fail to assess their capacity for iterative improvement under feedback. To bridge this gap, we propose a multi-round evaluation framework that introduces, for the first time, a research-process-oriented feedback mechanism and a Research Gap Inference (RGI) method based on scoring-criteria satisfaction patterns to identify strategic deficiencies and compare introspective versus process-oriented feedback. Experimental results show that process-level feedback improves normalized scores by 8–15 points in the first round with an adoption rate of 35%–40%; however, subsequent rounds exhibit regression in up to 24% of previously satisfied criteria due to rewriting, revealing that DRAs simultaneously demonstrate improvement and deterioration across iterations—challenging the assumption of their consistent optimization capability.

Abstract

Existing benchmarks for deep research agents (DRAs) assess only single-shot outputs, ignoring a key question: can DRAs improve their reports when guided by feedback? To investigate this, we conduct a multi-turn evaluation of DRAs under two feedback settings: self-reflection, in which the agent revises its report without any external diagnostic signal, and process-level feedback, in which the agent receives guidance targeting gaps in its research strategy. To enable process-level feedback, we design Research Gap Inference (RGI), a method that analyzes patterns of satisfied and unsatisfied rubric criteria to infer research-process gaps. Our analysis reveals three key findings: (i) under self-reflection, agents incorporate and regress on rubric criteria at nearly equal rates, yielding negligible net improvement; (ii) a single round of process-level feedback yields substantial gains, raising the normalized score by approximately $8$-$15$ points and yielding a roughly $35$-$40\%$ incorporation rate; (iii) these gains do not compound over subsequent turns, as agents regress on up to $24\%$ of previously satisfied criteria when rewriting the full report to address remaining gaps. Even with targeted guidance, reliable multi-turn improvement remains out of reach for the DRA architectures we evaluate. Our code and results are publicly available at https://github.com/sabharwalrishabh/Multi-Turn-Evaluation-of-DRAs.

Problem

• deep research agents
• multi-turn evaluation
• process-level feedback
• self-reflection
• research gap

Innovation

• multi-turn evaluation
• deep research agents
• process-level feedback
• Research Gap Inference
• self-reflection