Technology Category
Reasoning under Uncertainty: CausalityMachine Learning: Causal LearningKnowledge Representation and Reasoning: Computational Complexity of Reasoning
Are you doing better than random guessing? A call for using negative controls when evaluating causal discovery algorithms
🤖 AI Summary
Existing causal discovery algorithms lack standardized evaluation benchmarks; conventional metrics (e.g., precision, recall) are highly susceptible to random guessing—especially in sparse graphs—yielding false positive rates exceeding 0.8 and severely compromising performance assessment. Method: We propose a normalized evaluation paradigm using random guessing as a negative control, focusing on skeleton estimation. We formally establish, for the first time, a theoretically grounded negative-control benchmark for causal discovery evaluation; derive exact distributions of key metrics under the random-hypothesis null; develop a statistically principled skeleton-fitting significance test grounded in statistical inference and random graph theory; and extend the negative-control framework to full causal structure evaluation. Contribution/Results: Validated via Monte Carlo simulations and real biological datasets, our approach substantially improves assessment reliability. We publicly release an open-source evaluation pipeline to foster community-wide standardization.
Technology Category
Application Category
📝 Abstract
New proposals for causal discovery algorithms are typically evaluated using simulations and a few select real data examples with known data generating mechanisms. However, there does not exist a general guideline for how such evaluation studies should be designed, and therefore, comparing results across different studies can be difficult. In this article, we propose a common evaluation baseline by posing the question: Are we doing better than random guessing? For the task of graph skeleton estimation, we derive exact distributional results under random guessing for the expected behavior of a range of typical causal discovery evaluation metrics (including precision and recall). We show that these metrics can achieve very large values under random guessing in certain scenarios, and hence warn against using them without also reporting negative control results, i.e., performance under random guessing. We also propose an exact test of overall skeleton fit, and showcase its use on a real data application. Finally, we propose a general pipeline for using random controls beyond the skeleton estimation task, and apply it both in a simulated example and a real data application.
Problem
Research questions and friction points this paper is trying to address.
Lack of guidelines for evaluating causal discovery algorithms
Need for negative controls to assess random guessing performance
Proposing exact tests and pipelines for broader negative control use
Innovation
Methods, ideas, or system contributions that make the work stand out.
Using negative controls for evaluation baseline
Deriving exact distributional results under guessing
Proposing general pipeline for negative controls
🔎 Similar Papers
No similar papers found.
