version control

Reproducibility remains a critical challenge in large language model (LLM)-driven software engineering (SE) research, undermining credibility and cumulative scientific progress. Method: We conducted a systematic literature review of 640 papers, integrating structured metadata extraction, manual annotation, and cross-platform analysis to diagnose reproducibility deficiencies across code, data, execution environments, and version control. Contribution/Results: We propose a novel taxonomy of seven reproducibility defect categories and introduce the Reproducibility Maturity Model (RMM), shifting evaluation from binary “reproducible/not reproducible” to a multi-dimensional, incremental framework. Our findings reveal that even top-tier conferences’ artifact evaluation badges exhibit low enforcement fidelity and poor long-term reproducibility; publication venue transparency practices vary substantially. This work provides both a theoretical framework and empirical evidence to enhance the rigor and trustworthiness of LLM-SE research.

This study addresses the significant burden developers face in authoring and maintaining GitHub Actions workflows, stemming from a lack of systematic understanding of real-world automation and reuse practices. Through a mixed-methods approach combining a survey of 419 practitioners with qualitative and quantitative analysis, this work presents the first developer-centric characterization of common automation tasks, patterns of reuse mechanism adoption, and maintenance pain points in workflow development. The findings reveal that while developers heavily rely on reusable Actions, they seldom adopt reusable workflows; version management challenges lead to rampant copy-pasting; and critical aspects such as security and performance monitoring remain under-automated. These insights provide empirical foundations for improving CI/CD toolchains and reuse mechanisms.

Although Docker is widely assumed to ensure reproducibility of software environments, its practical efficacy remains insufficiently validated. This study presents the first systematic investigation combining a literature review with large-scale empirical analysis of 5,298 real-world GitHub projects. By reconstructing Docker images, performing differential comparisons, and mining workflow patterns, we quantitatively assess the reproducibility of Docker builds and the effectiveness of recommended best practices. Our findings reveal that a significant proportion of Docker builds are not reproducible, and existing best practices offer limited improvements in practice. These results challenge the prevailing assumption that “containers guarantee reproducibility” and provide empirical evidence and actionable insights for enhancing reproducibility in computational research.

This work addresses the limitations of existing reproducibility assessment methods, which rely on manual annotations and thus lack scalability and authentic supervision signals reflecting real-world reproduction challenges. The authors propose the first scalable evaluation framework that leverages GitHub user-submitted issues as natural supervision, enabling large-scale assessment of large language model (LLM) agents’ ability to identify paper-to-code reproducibility issues without human annotation. By integrating language understanding with code context analysis, the approach enables non-execution-based detection of reproducibility barriers. Experimental results demonstrate that the best-performing LLM agent identifies at least one semantically relevant reproducibility issue—aligned with those reported by humans—in approximately 90% of the evaluated papers, exhibiting strong performance in both failure detection and semantic localization.

Although top-tier conferences such as ICSE now commonly require authors to submit replication packages, the actual executability and reproducibility of these packages remain largely unassessed. This study presents a large-scale empirical investigation of 100 replication packages from ICSE papers published between 2015 and 2024, involving approximately 650 person-hours of manual execution, debugging, and root-cause analysis. The findings reveal that only 40% of the packages are executable, with just 32.5% running without modification; 82.5% require moderate to substantial changes. Among the executable packages, merely 35% successfully reproduce the original results. This work is the first to expose a significant gap between executability and reproducibility in software engineering replication packages and proposes three actionable guidelines to improve their reliability and utility.

Empirical software engineering faces significant challenges, including large-scale data, methodological complexity, and poor reproducibility, while the application of large language models (LLMs) in this domain lacks systematic integration. This study conducts a systematic literature review of 50 studies published between 2020 and 2025 across 12 leading conferences and journals, offering the first comprehensive taxonomy of 69 LLM-supported auxiliary tasks in empirical software engineering. The analysis reveals that LLMs are predominantly employed in data processing and analysis phases to enhance automation, efficiency, and scalability. However, their use is frequently hindered by issues such as hallucination, inconsistent outputs, sensitivity to prompting, insufficient reporting of reproducibility, and a notable absence of human-centered collaboration and transparency. Building on these findings, this work proposes a research agenda oriented toward the responsible application of LLMs in empirical software engineering.

Automatically reproducing executable bug-fix code pairs from unstructured developer Q&A posts is hindered by ambiguous descriptions and missing dependencies. This work proposes Reprodgen, the first end-to-end automated framework that leverages large language models to jointly model code intent (CI), functional requirements (FR), and structured chains of thought (SCoT) to generate semantically consistent and executable bug-fix code pairs. The approach incorporates an LLM-based iterative review mechanism coupled with real execution validation to ensure correctness. Evaluated on Stack Overflow and GitHub Issues across seven widely used data science libraries, the study introduces the first expert-validated, runnable benchmark of bug-fix pairs. Experimental results demonstrate that Reprodgen reliably reproduces code pairs exhibiting clear behavioral differences between buggy and fixed versions.

The practical feasibility of existing Sustainable Software Engineering (SSE) tools in regulated industries, such as finance, remains unclear. This study addresses this gap through a mixed-methods approach—combining an interactive web application taxonomy, qualitative workshops, and quantitative surveys—to systematically uncover developer preferences regarding ease of installation, minimal input requirements, and actionable outputs, including IDE/CI integration, dashboards, and automated refactoring suggestions. It further identifies key technical, organizational, and compliance-related factors influencing tool adoption. The findings provide empirical grounding for designing SSE tools tailored to regulated environments, highlighting how approval workflows and stringent compliance demands significantly constrain tool viability.

This work proposes the first deep integration of Git-style version control into MatrixOne, a cloud-native relational database, to overcome the limitations of existing version control systems in managing large-scale data and the lack of native support for branching, diffing, and merging in traditional databases. Leveraging MatrixOne’s immutable storage architecture and multi-version concurrency control (MVCC), the system enables near real-time cloning, branching/tagging, differencing, merging, and rollback operations on terabyte-scale datasets. It supports atomic deployments, isolation between development and production environments, and seamless integration into CI/CD pipelines. By ensuring strong consistency without service interruption, the approach significantly enhances collaboration efficiency and reliability in data engineering workflows.

This work addresses the frequent neglect of sampling strategy design and generalizability in software engineering research, which often undermines the representativeness of empirical findings. To remedy this, the paper introduces a domain-specific language (DSL) that explicitly models complex sampling workflows over code repositories through composable sampling operators, enabling—for the first time—formal specification and reasoning about the generalizability of sampling strategies. Implemented as a fluent Python API, the DSL is integrated with a statistical metric system to quantitatively assess the external validity of sampled datasets. The authors demonstrate the expressiveness and practical utility of their approach by reconstructing and formalizing the sampling procedures from multiple Mining Software Repositories (MSR) studies, thereby validating the framework’s capacity to capture real-world methodological diversity.