Model-Agnostic Feature Selection via LOCO-Guided Adaptive Minipatch Sampling

2026-09-21

AI Summary

本文提出LOCO引导自适应小批量采样(LAMPS)框架，解决黑盒模型特征选择难题，通过任何回归算法作为基础学习器，并结合观测和特征子抽样来识别重要特征。

Abstract

Black-box machine learning models increasingly deliver strong predictions, but extracting useful information from them, such as a set of important features, remains challenging. Existing model-agnostic methods primarily estimate feature importance or conduct inference on it rather than directly selecting features, whereas many feature selection methods are model-specific or rely on the model-X assumption. We introduce LOCO-guided Adaptive Minipatch Sampling (LAMPS), a model-agnostic ensemble framework that uses any black-box regression algorithm as its base learner to select features important for predicting the response. The base learner need only produce predictions and need not perform feature selection itself. LAMPS operates within a minipatch ensemble framework that subsamples both observations and features, allowing leave-one-covariate-out (LOCO) feature importance scores to be easily computed. It adaptively concentrates minipatch sampling on features with high LOCO scores while maintaining exploration. The resulting sampling probabilities rapidly separate signal from noise features after a few iterations, enabling selection through simple thresholding. We establish that LAMPS achieves exact feature selection in high-dimensional settings, provided that the base predictive models are sufficiently well trained on average. Extensive experiments on synthetic and real data show that LAMPS outperforms state-of-the-art feature selection methods, with particularly strong performance in the presence of correlated features.

Problem

• Black-box Models
• Feature Selection
• Model-agnostic Methods

Innovation

• Model-agnostic
• Feature Selection
• Adaptive Minipatch Sampling
• LOCO scores