Can AI agents conduct open-ended AI research? Early evidence from two case studies

2026-07-29

AI Summary

This study evaluates the capabilities of current AI agents in open-ended AI research, particularly in settings lacking clear validation criteria and high-quality peer review. To address this challenge, the authors introduce a novel “shadow evaluation” paradigm: AI agents independently reimplement the core research problems from two unpublished NeurIPS papers, and the original authors provide authoritative assessments of the agents’ outputs. The experiments employ state-of-the-art large language model agents, an autonomous engineering framework, and a multi-round experimental pipeline, consuming several thousand dollars of compute over six days. Results show that while the agents successfully completed all engineering implementation tasks, they failed to make substantive progress on the underlying research questions, leading the original authors to reject their submissions. The study identifies five systematic failure modes and fills a critical gap in evaluation methodologies for automated scientific research.

Abstract

Forecasts of explosive AI progress hinge on AI agents automating AI research. But evidence on whether agents can carry out open-ended AI research is thin. Current evaluations either test agents on narrow, verifiable tasks, which excludes open-ended research, or submit AI-generated papers to blind peer review, which is overstretched, stochastic, and suffers from poor review quality. We introduce a third way to measure progress towards AI R\&D automation. An agent takes on the central, open-ended research question of a high-quality unpublished paper, and the paper's original authors grade its output. We call these shadow evaluations. We ran shadow evaluations on two unpublished NeurIPS 2026 submissions, giving frontier agents six days and thousands of dollars of compute. The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions. As a result, both papers were unambiguously rejected by the authors. We identify five recurring failure modes: poor judgment about the bar for publishable research, uncreative responses to shortcomings in the research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift. A robustness check with a second model and scaffold reproduced these failures. We release the expert reviews, survey responses, agent repositories, and logs. Our results provide early evidence that today's agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle.

Problem

• AI agents
• open-ended research
• AI research automation
• shadow evaluation
• research lifecycle

Innovation

• shadow evaluation
• AI research automation
• open-ended research
• agent failure modes
• research lifecycle