An Agentic Approach Towards Replication Package Quality Evaluation

2026-06-01

AI Summary

This work addresses the inefficiency and limited scalability of manual evaluation of software engineering reproducibility packages by introducing, for the first time, a multi-agent architecture for assessing reproducibility quality. The authors translate open science guidelines into 31 machine-verifiable reproducibility criteria and integrate rule-based engines with automated scripts to perform both static and dynamic analyses of code, environments, and artifacts. The system generates evidence-based recommendations for improvement and supports human-in-the-loop optimization. Experimental evaluation on five reproducibility packages demonstrates 91.4% execution consistency and 75.4% detection accuracy, while user studies confirm its practical utility and strong potential for adoption.

Abstract

Reproducibility in empirical software engineering relies on complete, accessible, and reusable research artifacts, yet artifact evaluation remains largely manual and difficult to scale. This emerging results paper explores an agentic approach for assessing replication package quality by translating open-science guidelines into machine-verifiable criteria. We consolidate 380 requirements from 34 sources into 51 reproducibility criteria, of which 31 are operationalized for automated artifact-based evaluation. Based on these criteria, we implement a multi-agent prototype that automatically inspects replication packages and produces evidence-grounded improvement reports. A preliminary evaluation on five replication packages shows high inter-run consistency of 91.4\% and 75.4\% correctness, through micro-averaged agreement with a manual baseline. The agent performs best on structural criteria such as code, environment, and artifact availability, but struggles with qualitative or mixed-method studies. A pilot survey with seven software engineering researchers indicates well perceived usefulness and adoption potential, while revealing cognitive load in the human-in-the-loop planning step. Overall, these emerging results indicate that agentic research artifact evaluation has the potential to support authors and reviewers by automating selected routine checks.

Problem

• reproducibility
• artifact evaluation
• replication package
• software engineering
• open science

Innovation

• agentic evaluation
• replication package
• reproducibility criteria
• automated artifact assessment
• multi-agent system