Comparing Model-agnostic Feature Selection Methods through Relative Efficiency

2025-08-19

AI Summary

Model-agnostic feature selection suffers from high computational cost and insufficient theoretical foundations. This paper systematically compares two prominent approaches—Generalized Covariance Measure (GCM) and Leave-One-Covariate-Out (LOCO)—and, for the first time, derives their asymptotic relative efficiency within a unified framework under linear, nonlinear additive, and single-index models. Under standard regularity conditions, we theoretically establish that GCM is asymptotically more efficient than LOCO. Extensive simulations and real-data experiments—using black-box models including neural networks and gradient-boosted trees—robustly corroborate this theoretical finding. Our work provides the first rigorous theoretical justification for the efficiency advantage of GCM-based methods and establishes a new benchmark for model-agnostic feature importance assessment that is both statistically more efficient and inherently interpretable.

Technology Category

• Machine Learning: Dimensionality Reduction/Feature Selection
• Reasoning under Uncertainty: Graphical Models
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

Feature selection and importance estimation in a model-agnostic setting is an ongoing challenge of significant interest. Wrapper methods are commonly used because they are typically model-agnostic, even though they are computationally intensive. In this paper, we focus on feature selection methods related to the Generalized Covariance Measure (GCM) and Leave-One-Covariate-Out (LOCO) estimation, and provide a comparison based on relative efficiency. In particular, we present a theoretical comparison under three model settings: linear models, non-linear additive models, and single index models that mimic a single-layer neural network. We complement this with extensive simulations and real data examples. Our theoretical results, along with empirical findings, demonstrate that GCM-related methods generally outperform LOCO under suitable regularity conditions. Furthermore, we quantify the asymptotic relative efficiency of these approaches. Our simulations and real data analysis include widely used machine learning methods such as neural networks and gradient boosting trees.

Problem

• Evaluating relative efficiency of model-agnostic feature selection methods
• Comparing GCM and LOCO approaches under different model settings
• Quantifying asymptotic performance differences through theoretical and empirical analysis

Innovation

• Model-agnostic feature selection comparison
• Generalized Covariance Measure outperforms LOCO
• Theoretical efficiency analysis across models