A case study of evaluating AI agents on a neuroscience data-to-discovery pipeline

2026-06-05

AI Summary

Current AI systems struggle to autonomously execute end-to-end scientific discovery pipelines in neuroscience, particularly due to a lack of capability in assessing scientific plausibility. This work presents the first systematic evaluation of general-purpose code-generating agents on a real-world, large-scale optogenetics task in Drosophila, whose scale and complexity substantially exceed existing benchmarks. By integrating iterative code generation, visualization of intermediate outputs, and rigorous domain-expert-driven assessment, the study reveals that while agents can successfully complete individual pipeline stages, they fail to reliably orchestrate the full workflow. Performance degrades markedly in the absence of predefined evaluation criteria. These findings highlight critical limitations in AI’s capacity for scientific reasoning and self-evaluation, and propose principles for constructing and evaluating agent-based approaches to open-ended scientific problems.

Abstract

Agentic AI tools offer a promising path to automating software development bottlenecks in scientific research pipelines, particularly for stages that take domain experts days to months to build, where scientists care about correctness and robustness, not implementation details. We present an empirical study of general-purpose coding agents on a fly optogenetics data-to-discovery pipeline. We assess agents on tasks substantially larger than existing benchmarks, datasets orders of magnitude bigger, and evaluation criteria grounded in domain expert standards. We show that agents can solve several individual pipeline stages, suggesting stage-level automation is tractable. By analyzing agents' code iterations, we show that they struggle most when there is not a pre-defined criterion to iterate on, and they must instead use their scientific judgment to assess their current solution, a key open challenge. Mirroring scientific practice, they sometimes attempt visual inspection of intermediate outputs for self-evaluation, but largely fail to interpret what they see or act on it appropriately. Solving the end-to-end pipeline correctly requires stringing together successes across all pipeline stages, and this is beyond agents' current abilities. We identify challenges largely absent from existing benchmarks, including computational resource management and generalization to large held-out data collections. Finally, we distill principles for constructing scientific tasks and rigorous evaluation criteria for open-ended problems.

Problem

• AI agents
• scientific pipelines
• data-to-discovery
• evaluation criteria
• scientific judgment

Innovation

• agentic AI
• scientific automation
• data-to-discovery pipeline
• code evaluation
• neuroscience