configuration management

This study addresses the challenge of limited reproducibility and transparency in software engineering controlled experiments, often stemming from inadequate documentation. While generic preregistration templates—such as those provided by the Open Science Framework (OSF)—exist, they fail to comprehensively address the specific needs of software engineering research. This work presents the first systematic evaluation of the OSF preregistration template’s applicability to software engineering experiments, combining literature analysis, template comparison, and cross-referencing against established software engineering experimental reporting guidelines. The findings reveal that although existing OSF templates partially satisfy methodological requirements, none fully encompass all critical elements, and their customization capabilities are constrained. Based on these insights, the paper advocates for and provides a foundation toward developing a domain-specific, standardized preregistration template tailored to software engineering, thereby filling a critical gap in the field.

This work addresses the persistent challenge of inconsistent development and execution environments faced by researchers operating across heterogeneous computing platforms—ranging from laptops and workstations to supercomputers and cloud infrastructures. To overcome this, the authors propose a modular and portable software ecosystem featuring a unified command-line interface that enables seamless orchestration and execution of scientific workflows. The system ensures cross-platform consistency, reproducibility, and scalability, thereby streamlining computational research across diverse hardware configurations. Its practical efficacy has been demonstrated through successful integration into the plan4res project under the European Union’s Horizon 2020 initiative, where it effectively supported complex, large-scale scientific workflows in varied computing environments.

Addressing challenges in FAIR principle implementation—including fragmented data and code lifecycles, lack of executable environments, and high technical barriers—this study proposes a unified open-science platform. The platform uniquely integrates version control, containerized computational environments, and modular project scaffolding to support end-to-end reproducible research, from grant proposal to publication. It interoperates with mainstream scientific toolchains, supports deployment on both local workstations and institutional servers, and provides a lightweight graphical user interface. Empirical validation demonstrates successful re-execution of over a dozen interdisciplinary studies published more than ten years ago, confirming the platform’s robust long-term reproducibility, cross-platform compatibility, and seamless execution across diverse domains. By significantly lowering technical adoption barriers for researchers, the platform enables practical integration of FAIR principles and reproducibility practices into routine scientific workflows.

This paper addresses core challenges to computational experiment reproducibility in HPC and systems research—namely, strong hardware dependence, complex environment configuration, and high operational costs. Methodologically, it introduces a layered, pragmatic solution framework grounded in a novel “feasibility–rigor trade-off” principle, and proposes a dual-dimensional (challenge–recommendation) model tailored for authors, reviewers, institutions, and the broader community. The approach integrates structured workshops, cross-role consensus modeling, actionable checklists, and explores ecosystem innovations including AI-assisted environment generation and artifact digital repositories. Key contributions include: (1) a comprehensive, lifecycle-spanning practical guide and standardized reproducibility checklist; (2) a widely adopted community consensus on reproducible practices within the HPC domain; and (3) a systemic pathway advancing reproducibility from conceptual principle to institutionalized practice.

This work addresses the challenges researchers face in coordinating hardware control, data analysis, and experimental planning when building autonomous experimentation systems. To overcome these barriers, the authors propose a general-purpose, service-oriented autonomous experimentation platform featuring a language-agnostic, modular architecture. The platform enables users to define custom modules for hardware control, data processing, and experimental planning, with efficient inter-module communication facilitated through protobuf and gRPC. Integrated components—including a unified central control interface, automated UI generation, data management infrastructure, and experimental design tools—support closed-loop autonomous experimentation. By significantly lowering deployment complexity while enhancing flexibility and scalability, this approach allows researchers to concentrate on domain-specific scientific innovation rather than system integration overhead.

Existing agent evaluation benchmarks predominantly focus on virtual software interactions and fail to assess the multimodal interface coordination and feedback-driven parameter tuning required for scientific instrument control. This work introduces the first benchmark specifically designed for this domain, presenting a web-based, extensible, secure, and reproducible simulator suite encompassing eight instrument types and 96 subtasks that fully span the workflow from sample loading to result inspection. The benchmark supports flexible task configuration and execution-based evaluation, integrating vision-language models with a dedicated agent framework. Experimental results demonstrate that while current agents can handle structured GUI subtasks, they struggle significantly with feedback-driven operations and long-horizon workflows, thereby validating the benchmark’s necessity and its capacity to expose critical gaps in agent capabilities.

This work addresses the inefficiency in notebook-based distributed workflows, where minor modifications often trigger full re-execution, severely hindering iterative development and reproducibility. To overcome this limitation, the authors propose NBRewind, a system that, for the first time, enables fine-grained incremental execution and cross-platform portability while preserving reproducibility. NBRewind integrates a dual-kernel architecture—comprising auditing and replay components—with cell-level incremental checkpoints and inter-cell dataflow analysis. It further leverages standardized notebook packaging to facilitate efficient partial re-execution. Evaluation in real-world high-performance computing (HPC) scenarios demonstrates that NBRewind incurs minimal overhead for incremental checkpointing and substantially improves both execution efficiency and cross-site reproducibility.

This work addresses the widespread irreproducibility of academic Jupyter Notebooks caused by environment drift, missing dependencies, and implicit execution assumptions. The authors propose the first web-oriented, automated reproducibility engineering pipeline that systematically reconstructs and evaluates repository-level execution environments for notebooks hosted on GitHub. By leveraging dependency inference, auto-generated Docker containers, and isolated execution, the pipeline enables large-scale assessment of reproducibility. A novel four-category execution outcome framework is introduced to quantify reproduction fidelity. Evaluation on 443 real-world notebooks shows that containerization resolves 66.7% of dependency-related failures; however, only 46.3% achieve high output fidelity, demonstrating that while containerization is necessary, it is insufficient for bit-for-bit reproducibility. These findings underscore the critical need for systematic reproducibility evaluation in computational research.

This work addresses the limitations of existing reproducibility assessment methods, which rely on manual annotations and thus lack scalability and authentic supervision signals reflecting real-world reproduction challenges. The authors propose the first scalable evaluation framework that leverages GitHub user-submitted issues as natural supervision, enabling large-scale assessment of large language model (LLM) agents’ ability to identify paper-to-code reproducibility issues without human annotation. By integrating language understanding with code context analysis, the approach enables non-execution-based detection of reproducibility barriers. Experimental results demonstrate that the best-performing LLM agent identifies at least one semantically relevant reproducibility issue—aligned with those reported by humans—in approximately 90% of the evaluated papers, exhibiting strong performance in both failure detection and semantic localization.

This work addresses the inefficiency and limited scalability of manual evaluation of software engineering reproducibility packages by introducing, for the first time, a multi-agent architecture for assessing reproducibility quality. The authors translate open science guidelines into 31 machine-verifiable reproducibility criteria and integrate rule-based engines with automated scripts to perform both static and dynamic analyses of code, environments, and artifacts. The system generates evidence-based recommendations for improvement and supports human-in-the-loop optimization. Experimental evaluation on five reproducibility packages demonstrates 91.4% execution consistency and 75.4% detection accuracy, while user studies confirm its practical utility and strong potential for adoption.