Problem
Research questions and friction points this paper is trying to address.
cross-domain benchmark
end-to-end scientific workflows
workflow completion
scientific deliverables
evaluation
FrontierChallenge: Evaluating Scientific Workflow Completion
🤖 AI Summary
本文通过引入FrontierChallenge基准，评估科学工作流的完成情况，使用97个跨领域任务测试12种前沿模型，发现高部分得分和完成声明不能可靠表明任务完全交付。
📝 Abstract
Scientific agents increasingly analyze data, execute code, and produce research artifacts, yet most benchmarks emphasize final answers, isolated programs, or a single domain. We introduce FrontierChallenge, a cross-domain benchmark comprising 300 end-to-end scientific workflows. In this paper, we release and evaluate 97 of these tasks, spanning quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry/environment. Each task provides fixed inputs and specifies a bundle of required scientific deliverables. We evaluate twelve frontier models with three agent scaffolds. Pass Rate measures the fraction of tasks satisfying the full-completion criterion, while Avg. Score captures partial progress. Each of the best-performing configurations completed only 20 of the 97 released tasks, yielding a Pass Rate of 20.6%. Partial progress translated especially poorly into complete delivery in analytical chemistry and electrochemistry/environment: Avg. Scores reached 87.6 and 94.9, but the highest Pass Rates were only 4% and 0%. Among non-passing Claude Code trajectories, 75.5% still ended with language claiming completion. These findings show that neither high partial scores nor confident claims of completion reliably indicate that a scientific task has been fully delivered, highlighting the need to evaluate end-to-end workflow execution and the completeness of scientific deliverables together.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Cross-domain Benchmark
End-to-End Scientific Workflows
Workflow Execution Evaluation
Scientific Deliverables Completeness
🔎 Similar Papers
No similar papers found.
