Information-theoretic Quantification of High-order Feature Effects in Classification Problems

2025-07-06

AI Summary

Existing model-agnostic feature importance methods struggle to characterize high-order interactions and disentangle overlapping contributions among features. To address this, we propose a higher-order feature effect decomposition framework grounded in conditional mutual information (CMI), which rigorously decomposes feature contributions into three distinct information components: uniqueness, synergy, and redundancy. Our method employs a k-nearest-neighbor estimator to uniformly estimate CMI for mixed-type (discrete and continuous) variables, ensuring model-agnostic interpretability. On synthetic Gaussian and non-Gaussian datasets, the framework accurately recovers ground-truth interaction structures; it further demonstrates robustness on real-world TCGA-BRCA gene expression data. Results show that the approach effectively supports interaction-aware feature selection and interpretable modeling. By formalizing feature dependencies through an information-theoretic lens, our work establishes a novel paradigm for modeling higher-order statistical dependencies.

Technology Category

• Machine Learning: Dimensionality Reduction/Feature Selection
• Reasoning under Uncertainty: Graphical Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

Understanding the contribution of individual features in predictive models remains a central goal in interpretable machine learning, and while many model-agnostic methods exist to estimate feature importance, they often fall short in capturing high-order interactions and disentangling overlapping contributions. In this work, we present an information-theoretic extension of the High-order interactions for Feature importance (Hi-Fi) method, leveraging Conditional Mutual Information (CMI) estimated via a k-Nearest Neighbor (kNN) approach working on mixed discrete and continuous random variables. Our framework decomposes feature contributions into unique, synergistic, and redundant components, offering a richer, model-independent understanding of their predictive roles. We validate the method using synthetic datasets with known Gaussian structures, where ground truth interaction patterns are analytically derived, and further test it on non-Gaussian and real-world gene expression data from TCGA-BRCA. Results indicate that the proposed estimator accurately recovers theoretical and expected findings, providing a potential use case for developing feature selection algorithms or model development based on interaction analysis.

Problem

• Quantify high-order feature interactions in classification models
• Disentangle unique, synergistic, and redundant feature contributions
• Validate method on synthetic and real-world gene expression data

Innovation

• Extends Hi-Fi method with Conditional Mutual Information
• Uses kNN for mixed discrete and continuous variables
• Decomposes feature contributions into unique, synergistic, redundant