Shaky Structures: The Wobbly World of Causal Graphs in Software Analytics

2025-05-18

AI Summary

This paper identifies severe structural instability in causal graphs within software engineering (SE): identical SE data yield substantially different causal graphs—often with contradictory causal conclusions—when processed by different causal discovery algorithms or subjected to minor perturbations. Method: We systematically evaluate four representative algorithms—PC, FCI, GES, and LiNGAM—across 23 SE datasets, quantifying structural robustness via Jaccard similarity of edge sets. We conduct rigorous robustness experiments: cross-project, cross-version, bootstrap resampling, and parameter perturbation. Contribution/Results: Over 50% of edges vary across generation conditions; causal inferences for three canonical SE tasks—configuration selection, project management, and defect prediction—exhibit high inconsistency. This work provides the first systematic quantification of causal graph fragility in SE, establishes the necessity of “causal graph robustness testing,” and challenges prevailing practices in SE causal inference.

Technology Category

Reasoning under Uncertainty: CausalityMachine Learning: Causal LearningKnowledge Representation and Reasoning: Action, Change, and Causality

Application Category

Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphsWeb Mining and Content Analysis: Robustness and generalizability of Web mining methodsResponsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Causal graphs are widely used in software engineering to document and explore causal relationships. Though widely used, they may also be wildly misleading. Causal structures generated from SE data can be highly variable. This instability is so significant that conclusions drawn from one graph may be totally reversed in another, even when both graphs are learned from the same or very similar project data. To document this problem, this paper examines causal graphs found by four causal graph generators (PC, FCI, GES, and LiNGAM) when applied to 23 data sets, relating to three different SE tasks: (a) learning how configuration options are selected for different properties; (b) understanding how management choices affect software projects; and (c) defect prediction. Graphs were compared between (a) different projects exploring the same task; (b) version i and i + 1 of a system; (c) different 90% samples of the data; and (d) small variations in the causal graph generator. Measured in terms of the Jaccard index of the number of edges shared by two different graphs, over half the edges were changed by these treatments. Hence, we conclude two things. Firstly, specific conclusions found by causal graph generators about how two specific variables affect each other may not generalize since those conclusions could be reversed by minor changes in how those graphs are generated. Secondly, before researchers can report supposedly general conclusions from causal graphs (e.g.,"long functions cause more defects"), they should test that such conclusions hold over the numerous causal graphs that might be generated from the same data.

Problem

• Causal graphs in software analytics are unstable and misleading
• Different graph generators produce highly variable causal structures
• Conclusions from causal graphs may not generalize due to instability

Innovation

• Compares four causal graph generators
• Tests stability across 23 datasets
• Validates conclusions over multiple graphs