MDArena: Evaluating Coding Agents on Realistic Molecular Dynamics Workflows

2026-07-31

AI Summary

This study addresses the lack of systematic evaluation of code-generating agents in authentic molecular dynamics (MD) research workflows. The authors introduce MDArena, the first benchmark tailored to real-world MD scenarios, encompassing 29 molecular systems, 14 research protocols, and 50 containerized tasks spanning critical stages such as trajectory analysis, system setup, and free energy calculations. Using strict pass rates (Strict-Pass@1), process-based rewards, and correctness analyses, they conduct the first comprehensive assessment of Codex and OpenCode family models. Results reveal that even the best-performing model (Codex GPT-5.5) achieves only a 48% pass rate, with most agents consistently failing on complex tasks like membrane protein setup and free energy protocol configuration, highlighting substantial deficiencies in fine-grained scientific reproducibility.

Abstract

Accelerating scientific discovery is among the most consequential applications of AI, and computational biomolecular simulation stands out as a particularly promising target within this broader effort. Coding agents promise to automate significant portions of this workflow, yet their reliability on realistic molecular dynamics (MD) tasks remains poorly characterized. To address this issue, we introduce MDArena, a benchmark of 50 containerized tasks drawn from active biomolecular simulation projects, spanning 29 molecular systems and 14 broad research protocols, including trajectory analysis, complex system preparation, free-energy protocols, and enhanced sampling. We evaluate six model/harness configurations spanning Codex and OpenCode. Among the evaluated configurations, Codex GPT-5.5 at extra-high reasoning effort performs best, reaching 24/50 Strict-Pass@1 successes (48%), followed by Codex GPT-5.5 Medium with 21/50, and OpenCode Gemini Flash 3.5 with 20/50. Average correctness and process rewards are substantially higher than strict success rates across all configurations, indicating that agents frequently make meaningful partial progress but fail on the fine-grained details required for reproducible scientific workflows. Hard tasks remain largely unsolved, particularly membrane-protein system preparation and alchemical free-energy setup, both unsolved or near-unsolved by every evaluated configuration. MDArena thus exposes a substantial gap between the usefulness of coding agents as supervised assistants and their reliability as autonomous MD researchers, while providing a reproducible and extensible platform for tracking progress toward closing it.

Problem

• coding agents
• molecular dynamics
• scientific workflows
• reliability
• benchmarking

Innovation

• MDArena
• coding agents
• molecular dynamics
• benchmarking
• autonomous scientific workflows