ECLAIR: A Causally-Grounded AI Framework for Scientific Discovery in Empirical Software Engineering

2026-08-03

AI Summary

This work addresses the challenge posed by the complexity of modern software systems, which hinders the systematic application of the scientific method in empirical software engineering. We propose a structured methodology for scientific discovery in this domain, uniquely integrating large language models as active agents that adhere to principles of causal reasoning throughout the entire scientific workflow. The framework incorporates human-in-the-loop mechanisms to ensure the reliability of inferential processes and supports hypothesis generation, experimental analysis, and result interpretation. Through a case study, we identify a weak yet statistically significant negative causal effect of prompt design on code generation accuracy, thereby demonstrating the efficacy of our approach in uncovering causal mechanisms underlying software phenomena.

Abstract

The scientific method has long guided empirical research in Software Engineering (SE), but the complexity of modern software systems often hinders its systematic application. This paper introduces _ECLAIR_, a causally grounded AI framework that integrates Large Language Models (_LLMs_) into every stage of the scientific process, from hypothesis generation to analysis and interpretation. _ECLAIR_ treats _LLMs_ as active **scientific agents** operating under the principles of causal inference, within a human-in-the-loop design that safeguards against the risks of unsound automated reasoning. We demonstrate the framework through a case study examining how prompt design influences code generation accuracy in two _LLMs_. Results show that, for both models, instruction-style, longer few-shot, and signature-augmented prompts yield small negative causal effects on accuracy, illustrating how causal reasoning provides a principled foundation for explaining _why_ software phenomena occur. This study presents the first causally grounded structured methodology for embedding _LLMs_ within the scientific method in SE, designed around the epistemological demands of empirical SE research, establishing a basis for rigorous AI-assisted research.

Problem

• scientific discovery
• empirical software engineering
• causal inference
• large language models
• AI-assisted research

Innovation

• causal inference
• large language models
• scientific method
• empirical software engineering
• human-in-the-loop