Let's Talk About It: Making Scientific Computational Reproducibility Easy

2025-04-14

AI Summary

Reproducing scientific computing results remains challenging due to missing experimental documentation, configuration details, and datasets—undermining result credibility. To address this, we propose the first natural language interaction paradigm explicitly designed for computational reproducibility: users describe experiments in plain English; the system automatically parses intent, infers dependencies, constructs a containerized execution environment, and packages code, data, and runtime into a single, cross-platform executable file enabling zero-configuration, one-click re-execution. Our approach integrates natural language understanding (NLU), automated environment provisioning, and lightweight container encapsulation. Evaluated on 18 published computational experiments, it achieves a high rate of fully automated, intervention-free reproduction. A user study demonstrates significantly higher System Usability Scale (SUS) scores compared to leading commercial tools and substantially lower NASA-TLX cognitive workload—validating simultaneous advances in usability and practical utility.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Natural language understanding for Web search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Computational reproducibility of scientific results, that is, the execution of a computational experiment (e.g., a script) using its original settings (data, code, etc.), should always be possible. However, reproducibility has become a significant challenge, as researchers often face difficulties in accurately replicating experiments due to inconsistencies in documentation, setup configurations, and missing data. This lack of reproducibility may undermine the credibility of scientific results. To address this issue, we propose a conversational, text-based tool that allows researchers to easily reproduce computational experiments (theirs or from others) and package them in a single file that can be re-executed with just a double click on any computer, requiring the installation of a single widely-used software. Researchers interact with the platform in natural language, which our tool processes to automatically create a computational environment able to execute the provided experiment/code. We conducted two studies to evaluate our proposal. In the first study, we gathered qualitative data by executing 18 experiments from the literature. Although in some cases it was not possible to execute the experiment, in most instances, it was necessary to have little or even no interaction for the tool to reproduce the results. We also conducted a user study comparing our tool with an enterprise-level one. During this study, we measured the usability of both tools using the System Usability Scale (SUS) and participants' workload using the NASA Task Load Index (TLX). The results show a statistically significant difference between both tools in favor of our proposal, demonstrating that the usability and workload of our tool are superior to the current state of the art.

Problem

• Addressing challenges in computational reproducibility of scientific experiments
• Proposing a tool to simplify experiment replication with natural language interaction
• Improving usability and reducing workload in computational reproducibility tools

Innovation

• Conversational tool for computational reproducibility
• Single-file packaging for easy re-execution
• Natural language processing for environment setup