experimental reproducibility

Scientific computing artifacts—such as analysis scripts and software prototypes—frequently suffer from poor reproducibility due to environmental heterogeneity, dependency drift, and inadequate documentation, thereby undermining research credibility. To address this, we introduce the first cross-disciplinary, structured, and standardized benchmark dataset for computational experiments, encompassing workflows ranging from single-script executions to multi-language, complex pipelines. Our framework uniformly models metadata, standardizes dependency declarations (e.g., requirements.txt, Dockerfiles), encapsulates multi-language execution procedures, and prescribes a rigorous documentation protocol. The dataset comprises dozens of human-validated, fully reproducible experimental cases, enabling objective, comparable, and reproducible evaluation of reproducibility tools. This work fills a critical gap in the field by providing the first systematic, community-grounded benchmark for assessing computational reproducibility, thereby significantly enhancing the rigor, transparency, and comparability of reproducibility research.

Current research on automated reproducibility lacks a unified problem formulation, hindering the development of general-purpose methods. This work proposes a general framework grounded in the scientific method, offering the first formal definition of reproducibility applicable to any empirical AI study. The framework leverages automated information extraction techniques to structurally identify and extract core scientific elements—such as hypotheses, experiments, and interpretations—from research papers. To evaluate its effectiveness, the authors construct a reproducibility dataset comprising 20 diverse studies. The majority of original authors affirm that the extracted content accurately represents their work, demonstrating the framework’s strong applicability and practical utility across varied AI subfields.

Experimental reproducibility in Empirical Software Engineering (ESE) is hindered by a fundamental disconnect between idealized methodological assumptions—e.g., standardized protocols and controlled conditions—and researchers’ actual experimental practices. Method: We conducted a two-year ethnographic study involving participant observation, in-depth interviews, and content analysis of experimental artifacts across diverse ESE research teams. Contribution/Results: We identify four critical dimensions—activity diversity, role distribution, conceptual granularity, and domain perspective—in which real-world experimentation systematically deviates from textbook models. Based on these findings, we propose the first high-fidelity conceptual and process model grounded in empirical research practice, explicitly capturing the “practice gap” underlying irreproducibility. This model provides foundational evidence and design principles for developing next-generation reproducibility-support tools, methodological guidelines, and evaluation frameworks in ESE.

The AI/ML community faces a severe reproducibility crisis, primarily driven by conceptual ambiguity in verification terminology—such as “reproducibility,” “replicability,” and “dependency/independence”—which undermines research credibility and scientific progress. To address this, we propose the first five-dimensional verification taxonomy, systematically defining core concepts—including reproducibility, dependency vs. independent re-executability, and direct vs. conceptual replicability—by clarifying their objectives, prerequisites, and evaluation criteria. Our framework integrates conceptual analysis, terminological standardization, and methodological modeling to yield a structured verification guideline. It enhances experimental rigor in study design, fosters consensus across the research community on verification practices, and significantly improves cross-team result reproducibility and outcome reliability.

This study addresses the persistent challenge of irreproducibility in medical AI research, often stemming from reliance on proprietary data and absent code, which undermines scientific credibility and clinical translation. Through a combination of bibliometric analysis, reproducibility audits, and evaluation of open-source practices, this work quantifies—for the first time—that papers releasing both data and code receive, on average, 110% more citations. Building on these findings, the study proposes a novel pathway centered on open-source software to standardize preprocessing pipelines and establish robust benchmarks. This approach offers empirical evidence and a practical framework for developing medical AI systems that are safe, effective, and trustworthy, thereby advancing both scientific rigor and real-world applicability in healthcare AI.

Traditional benchmarks provide only aggregate scores, offering insufficient evidence to support reliable deployment decisions and thereby creating a disconnect between evaluation and action. To address this gap, this work proposes a “deployment-completeness” benchmarking framework, introducing novel metrics—evidence fibers, completeness curves, and certifiable proportions—alongside a systematic audit methodology comprising evidence fiber analysis, response ranking intervals, conformal coverage evaluation, and a certify-then-acquire decision pipeline. Empirical evaluation on benchmarks such as Tox21, Matbench, and JARVIS reveals that conventional approaches suffer a drastic drop in channel coverage to 10.07% under real-world deployment conditions. In contrast, the proposed method reduces error-driven deployment decisions to 0.027% on Tox21 and 0.128% on JARVIS, substantially enhancing deployment reliability.

This work proposes an AI agent–driven workflow to address the high costs of reproducing large-scale empirical studies, which often stem from discrepancies in computational environments, code, and documentation. The approach decouples scientific reasoning from computational execution: researchers supply standardized diagnostic templates, and the system automatically retrieves and orchestrates reproduction materials within a version-controlled environment. A structured knowledge layer captures failure patterns, enabling adaptive reproduction across heterogeneous studies while ensuring transparency and stability of the analytical pipeline. Evaluated on 92 instrumental variable studies, the method achieves an 87% end-to-end reproduction success rate; when data and code are available, it attains 100% success at both the paper and model levels.

This work addresses the widespread irreproducibility of academic Jupyter Notebooks caused by environment drift, missing dependencies, and implicit execution assumptions. The authors propose the first web-oriented, automated reproducibility engineering pipeline that systematically reconstructs and evaluates repository-level execution environments for notebooks hosted on GitHub. By leveraging dependency inference, auto-generated Docker containers, and isolated execution, the pipeline enables large-scale assessment of reproducibility. A novel four-category execution outcome framework is introduced to quantify reproduction fidelity. Evaluation on 443 real-world notebooks shows that containerization resolves 66.7% of dependency-related failures; however, only 46.3% achieve high output fidelity, demonstrating that while containerization is necessary, it is insufficient for bit-for-bit reproducibility. These findings underscore the critical need for systematic reproducibility evaluation in computational research.

This work addresses the absence of end-to-end, reliable evaluation frameworks for autonomous scientific research systems in the context of machine learning reproducibility. It introduces MLReplicate, the first standardized benchmark for this task, constructed from outstanding papers accepted at ICML 2025. The authors employ a dual-track evaluation protocol combining automated conference-style peer review with structured expert assessment to systematically evaluate six state-of-the-art systems on their ability to generate complete scientific manuscripts. Results reveal that only 10 out of 45 generated manuscripts passed automated review, and 59% of those deemed acceptable contained fabricated or unsupported claims. Notably, workflow design proved more critical than computational scale: the system with the lowest resource consumption outperformed the highest-cost system in human evaluations despite a 38-fold difference in token usage, exposing widespread methodological flaws and hallucination in current approaches.

This study addresses a critical gap in existing scientific data analysis benchmarks, which fail to differentiate models’ capabilities across distinct scientific reasoning tasks—such as hypothesis exploration, causal inference, and mechanistic explanation. To this end, the authors introduce SDABench, the first multidimensional evaluation benchmark specifically designed to assess scientific analytical competence. It encompasses six dimensions: descriptive, exploratory, inferential, predictive, causal, and mechanistic reasoning, comprising 527 real-world and 6,000 synthetically generated data instances across five scientific domains. Using a five-stage error analysis framework, the benchmark systematically evaluates 15 prominent large language models. Results reveal strong performance on descriptive tasks but substantial deficiencies in complex reasoning involving hypothesis selection, latent variable modeling, and mechanistic inference, indicating that current models remain ill-equipped to support high-level scientific discovery.