Feature Bagging Provides Stability

2026-07-29

AI Summary

This work addresses the problem of quantifying and enhancing the stability of machine learning models under feature perturbations to improve generalization. It introduces, for the first time, a measure termed Feature Instability (FI), which captures complementary generalization information relative to instance-level instability. Building upon algorithmic stability theory, the study analyzes a feature bagging mechanism and establishes theoretical guarantees—under both parametric linear and model-free settings—that this mechanism effectively reduces FI. Empirical results demonstrate that feature bagging significantly lowers FI, with only a few iterations needed to approach the stability achieved by infinite bagging; moreover, aggressive feature subsampling yields further improvements in stability.

Abstract

We study feature bagging through the lens of algorithmic stability. Feature bagging is an ensemble strategy that aggregates base learners trained on randomly subsampled feature subsets, possibly in a data-dependent manner. We introduce feature instability (FI), the feature-axis analogue of instance instability (II), which measures sensitivity to removing a single feature. Smaller values of II or FI correspond to stronger stability, and our experiments show that FI captures generalization-relevant information complementary to II. Within this framework, we analyze feature bagging in both a parametric linear model and a model-free setting inspired by recursive feature subsampling in random forests. In both settings, we establish formal guarantees showing that feature bagging improves the relevant stability relative to its non-bagged counterpart, with larger improvements under more aggressive subsampling. We further show that a modest number of bagging rounds is sufficient to approach the infinite-bagging stability level.

Problem

• feature bagging
• algorithmic stability
• feature instability
• generalization
• ensemble methods

Innovation

• feature bagging
• algorithmic stability
• feature instability
• ensemble learning
• generalization