open-source release

Although top-tier conferences such as ICSE now commonly require authors to submit replication packages, the actual executability and reproducibility of these packages remain largely unassessed. This study presents a large-scale empirical investigation of 100 replication packages from ICSE papers published between 2015 and 2024, involving approximately 650 person-hours of manual execution, debugging, and root-cause analysis. The findings reveal that only 40% of the packages are executable, with just 32.5% running without modification; 82.5% require moderate to substantial changes. Among the executable packages, merely 35% successfully reproduce the original results. This work is the first to expose a significant gap between executability and reproducibility in software engineering replication packages and proposes three actionable guidelines to improve their reliability and utility.

This work addresses the lack of effective mechanisms to verify the consistency between distributed software artifacts and their source code in large-scale software distribution. To this end, we propose Lila, the first decentralized, reproducible build monitoring framework tailored to functional package management models. By aggregating distributed build reports into a reproducibility database, Lila enables continuous monitoring of over 80,000 software packages and sustains a reproducible build rate exceeding 90%. Our system not only fills a critical gap in reproducibility monitoring within functional package ecosystems but also bridges academic research and real-world deployment needs by delivering a scalable, decentralized infrastructure.

Although Docker is widely assumed to ensure reproducibility of software environments, its practical efficacy remains insufficiently validated. This study presents the first systematic investigation combining a literature review with large-scale empirical analysis of 5,298 real-world GitHub projects. By reconstructing Docker images, performing differential comparisons, and mining workflow patterns, we quantitatively assess the reproducibility of Docker builds and the effectiveness of recommended best practices. Our findings reveal that a significant proportion of Docker builds are not reproducible, and existing best practices offer limited improvements in practice. These results challenge the prevailing assumption that “containers guarantee reproducibility” and provide empirical evidence and actionable insights for enhancing reproducibility in computational research.

Reproducibility remains a critical challenge in large language model (LLM)-driven software engineering (SE) research, undermining credibility and cumulative scientific progress. Method: We conducted a systematic literature review of 640 papers, integrating structured metadata extraction, manual annotation, and cross-platform analysis to diagnose reproducibility deficiencies across code, data, execution environments, and version control. Contribution/Results: We propose a novel taxonomy of seven reproducibility defect categories and introduce the Reproducibility Maturity Model (RMM), shifting evaluation from binary “reproducible/not reproducible” to a multi-dimensional, incremental framework. Our findings reveal that even top-tier conferences’ artifact evaluation badges exhibit low enforcement fidelity and poor long-term reproducibility; publication venue transparency practices vary substantially. This work provides both a theoretical framework and empirical evidence to enhance the rigor and trustworthiness of LLM-SE research.

This study addresses the challenge of limited reproducibility and transparency in software engineering controlled experiments, often stemming from inadequate documentation. While generic preregistration templates—such as those provided by the Open Science Framework (OSF)—exist, they fail to comprehensively address the specific needs of software engineering research. This work presents the first systematic evaluation of the OSF preregistration template’s applicability to software engineering experiments, combining literature analysis, template comparison, and cross-referencing against established software engineering experimental reporting guidelines. The findings reveal that although existing OSF templates partially satisfy methodological requirements, none fully encompass all critical elements, and their customization capabilities are constrained. Based on these insights, the paper advocates for and provides a foundation toward developing a domain-specific, standardized preregistration template tailored to software engineering, thereby filling a critical gap in the field.

This work addresses the inefficiency and limited scalability of manual evaluation of software engineering reproducibility packages by introducing, for the first time, a multi-agent architecture for assessing reproducibility quality. The authors translate open science guidelines into 31 machine-verifiable reproducibility criteria and integrate rule-based engines with automated scripts to perform both static and dynamic analyses of code, environments, and artifacts. The system generates evidence-based recommendations for improvement and supports human-in-the-loop optimization. Experimental evaluation on five reproducibility packages demonstrates 91.4% execution consistency and 75.4% detection accuracy, while user studies confirm its practical utility and strong potential for adoption.

This study addresses the widespread practice of directly copying open-source code to bypass dependency management, which obscures license compliance risks. Leveraging the World of Code dataset, the authors construct a code reuse network through large-scale clone detection and quantify, for the first time at the scale of the entire open-source ecosystem, the compliance risks arising from such copy-paste reuse. Their analysis reveals that 39.4% of project compositions entail potential license conflicts, yet conventional dependency analysis tools capture only 2.43% of these instances, indicating severe under-detection. Integrating network modeling and regression analysis, the study further finds that code under permissive licenses such as MIT and Apache is reused across programming languages more frequently, whereas public-domain-licensed code exhibits comparatively lower reuse rates.

Addressing challenges in FAIR principle implementation—including fragmented data and code lifecycles, lack of executable environments, and high technical barriers—this study proposes a unified open-science platform. The platform uniquely integrates version control, containerized computational environments, and modular project scaffolding to support end-to-end reproducible research, from grant proposal to publication. It interoperates with mainstream scientific toolchains, supports deployment on both local workstations and institutional servers, and provides a lightweight graphical user interface. Empirical validation demonstrates successful re-execution of over a dozen interdisciplinary studies published more than ten years ago, confirming the platform’s robust long-term reproducibility, cross-platform compatibility, and seamless execution across diverse domains. By significantly lowering technical adoption barriers for researchers, the platform enables practical integration of FAIR principles and reproducibility practices into routine scientific workflows.

This work addresses the limitations of existing reproducibility assessment methods, which rely on manual annotations and thus lack scalability and authentic supervision signals reflecting real-world reproduction challenges. The authors propose the first scalable evaluation framework that leverages GitHub user-submitted issues as natural supervision, enabling large-scale assessment of large language model (LLM) agents’ ability to identify paper-to-code reproducibility issues without human annotation. By integrating language understanding with code context analysis, the approach enables non-execution-based detection of reproducibility barriers. Experimental results demonstrate that the best-performing LLM agent identifies at least one semantically relevant reproducibility issue—aligned with those reported by humans—in approximately 90% of the evaluated papers, exhibiting strong performance in both failure detection and semantic localization.

This study addresses the critical issue of declining reproducibility in quantum software defect datasets—such as Bugs4Q—due to dependency evolution, which undermines research reliability. The authors present the first systematic evaluation of this reproducibility degradation by reproducing 37 bugs across 21 Qiskit versions through 77,700 executions. Combining root cause analysis, dependency management, and API migration insights, they demonstrate that 93.6% of reproduction failures stem from environmental dependency issues rather than actual bug disappearance. Based on these findings, they propose a novel maintenance paradigm requiring source-level fixes and introduce an enhanced dataset, Bugs4Q-Robust, which boosts the reproduction rate from 16.2% to 78.4% on Qiskit v2.3.1—substantially outperforming conventional version-locking approaches.