🤖 AI Summary
Conventional binary classification metrics (e.g., AUROC) suffer from representational bias under label-local exchange invariance (LxC), rendering them inadequate for LxC-invariant tasks such as causal discovery.
Method: We propose LxCIM, the first evaluation metric explicitly designed for LxC invariance. We formally define LxC invariance and construct a unified framework satisfying ranking consistency, LxC invariance, computability, and multi-faceted interpretability. Leveraging insights from ROC and decision rate curves, we introduce the Cumulative Accuracy–Decision Rate Curve (CADRC) for fine-grained model behavior analysis. We rigorously establish theoretical connections between LxCIM and AUROC, accuracy, and AUDRC.
Results: Experiments on bivariate causal discovery demonstrate that LxCIM significantly outperforms mainstream baselines, effectively mitigating evaluation bias. The implementation is open-sourced, confirming its robustness and practical utility.
📝 Abstract
Binary classification is one of the oldest, most prevalent, and studied problems in machine learning. However, the metrics used to evaluate model performance have received comparatively little attention. The area under the receiver operating characteristic curve (AUROC) has long been a standard choice for model comparison. Despite its advantages, AUROC is not always ideal, particularly for problems that are invariant to local exchange of classes (LxC), a new form of metric invariance introduced in this work. To address this limitation, we propose LxCIM (LxC-invariant metric), which is not only rank-based and invariant under local exchange of classes, but also intuitive, logically consistent, and always computable, while enabling more detailed analysis through the cumulative accuracy-decision rate curve. Moreover, LxCIM exhibits clear theoretical connections to AUROC, accuracy, and the area under the accuracy-decision rate curve (AUDRC). These relationships allow for multiple complementary interpretations: as a symmetric form of AUROC, a rank-based analogue of accuracy, or a more representative and more interpretable variant of AUDRC. Finally, we demonstrate the direct applicability of LxCIM to the bivariate causal discovery problem (which exhibits invariance to local exchange of classes) and show how it addresses the acknowledged limitations of existing metrics used in this field. All code and implementation details are publicly available at github.com/tiagobrogueira/Causal-Discovery-In-Exchangeable-Data.