Towards Improving the External Validity of Software Engineering Experiments with Transportability Methods

2026-04-09

AI Summary

This study addresses the limited external validity of software engineering experiments, often stemming from unrepresentative samples. It pioneers the systematic application of causal inference–based transportability methods in this domain, integrating experimental and observational data to develop tailored implementation pathways and practical guidelines. The proposed approach is validated through simulation studies and offers actionable strategies for generalizing findings from common yet constrained settings—such as using students as proxies for professional developers—to broader target populations. By explicitly modeling the mechanisms underlying population differences, the method significantly enhances the practical applicability and reliability of experimental results across diverse real-world contexts.

Abstract

Controlled experiments are a core research method in software engineering (SE) for validating causal claims. However, recruiting a sample of participants that represents the intended target population is often difficult or expensive, which limits the external validity of experimental results. At the same time, SE researchers often have access to much larger amounts of observational than experimental data (e.g., from repositories, issue trackers, logs, surveys and industrial processes). Transportability methods combine these data from experimental and observational studies to "transport" results from the experimental sample to a broader, more representative sample of the target population. Although the ability to combine observational and experimental data in a principled way could substantially benefit empirical SE research, transportability methods have - to our knowledge - not been adopted in SE. In this vision, we aim to help make that adoption possible. To that end, we introduce transportability methods, their prerequisites, and demonstrate their potential through a simulation. We then outline several SE research scenarios in which these methods could apply, e.g., how to effectively use students as substitutes for developers. Finally, we outline a road map and practical guidelines to support SE researchers in applying them. Adopting transportability methods in SE research can strengthen the external validity of controlled experiments and help the field produce results that are both more reliable and more useful in practice.

Problem

• external validity
• software engineering experiments
• transportability
• observational data
• experimental data

Innovation

• transportability
• external validity
• software engineering experiments
• observational data
• causal inference