SciCode-Verified: How Benchmark Defects Underestimated the Scientific-Coding Ability of Language Models

2026-08-05

AI Summary

This study addresses a critical underestimation of large language models’ scientific programming capabilities in the original SciCode benchmark, attributable to flawed scoring mechanisms. Through systematic auditing by domain experts across all 65 problems, the authors identify and correct 263 errors, establishing SciCode-Verified—a high-quality, reproducible benchmark. Their methodology integrates dual-expert cross-validation, problem specification refinement, tolerance calibration, and scoring logic repair, supported by numerical experiments and theoretical analysis to ensure rigor. Post-correction, model accuracy on subproblems improves from 45–60% to 84–98%, and main-problem accuracy surges from 9–27% to 69–92%, revealing the models’ true performance. This work uncovers deep-seated scoring issues in scientific programming evaluation that are only detectable with domain expertise and introduces a traceable correction framework alongside refined evaluation standards.

Abstract

SciCode is the standard measure of the scientific-coding ability of language models: research-level problems that demand both frontier scientific theory and its implementation as working numerical code. It is a component of the Artificial Analysis Intelligence Index and a standing evaluation in government and national-laboratory suites. Yet its scores have recently plateaued: the strongest 2026 models cluster tightly around 60\% subproblem accuracy, and a successor model ties its predecessor. We trace this stagnation to defects in the benchmark itself. A per-problem, domain-expert audit of all 65 test problems uncovers 263 defects; 192 of them, spread across 91\% of the main problems, cause correct, instruction-following solutions to be wrongly rejected---through non-reproducible gold answers, over-tight tolerances, or self-contradictory specifications. Critically, 78\% of these score-suppressing defects require specialized physics or mathematics knowledge to detect, not mere clerical proofreading. We corrected every confirmable defect to produce SciCode-Verified. The corrections add only the specifications a well-posed problem requires, repair grading, and tighten the tests that were too lenient; every change is recorded with its justification and independently re-checked by a second domain expert. We re-evaluate twelve frontier model snapshots on the corrected benchmark and find a substantial recovery: subproblem accuracy rises from 45--60\% to 84--98\%, and main-problem accuracy from 9--27\% to 69--92\%. State-of-the-art models are far more proficient in scientific coding than SciCode has suggested---the bottleneck was not model capability, but the quality of the evaluation instrument. We release SciCode-Verified with its complete audit trail as the corrected public standard.

Problem

• scientific coding
• benchmark defects
• language models
• evaluation bias
• SciCode

Innovation

• benchmark auditing
• scientific coding
• language models
• evaluation reliability
• domain-expert verification