🤖 AI Summary
This work addresses the susceptibility of feature selection on small-to-medium scientific datasets to instability and optimistic bias induced by data leakage, which mutually exacerbate one another. The authors propose the first unified framework that integrates bootstrap-based stability selection with rigorous nested cross-validation, performing preprocessing and feature selection entirely within each inner fold to simultaneously yield highly stable feature subsets and unbiased performance estimates. Centered on stability as the primary optimization objective, the method supports binary classification, multiclass classification, and regression tasks, incorporates nine diverse algorithms, and includes a deterministic test suite to ensure reproducibility. Evaluated on three real-world datasets, the approach achieves predictive performance comparable to the best baselines while demonstrating significantly higher Jaccard stability than ANOVA F-test, recursive feature elimination, and Boruta.
📝 Abstract
Small-to-medium scientific datasets place machine learning pipelines under two compounding pressures. Single-run feature selection produces feature sets that change substantially under small perturbations of the training data, and any procedure that uses the same data for selection, tuning, and evaluation produces optimistically biased performance estimates. The two failure modes are routinely treated as separable, but in the regimes where scientific data live, they interact: an unstable selection inflates the variance of an already-optimistic score, and standard remedies for one rarely address the other. RobustModelMaker is a Python framework that couples bootstrap stability selection with strict nested cross-validation, performs all preprocessing and selection inside each fold, and produces a stability-tested feature subset together with a leakage-safe performance estimate. The framework supports nine algorithms across binary classification, multiclass classification, and regression. Behaviour is verified by a deterministic test suite spanning unit, performance, and reproducibility checks on three real scientific datasets comparing to three alternative selectors (ANOVA F-test, recursive feature elimination with cross-validation, and Boruta) on both predictive score and a Jaccard measure of selection stability. RobustModelMaker is competitive in score with the best alternative selector on each dataset, and occupies a position on the joint score-stability frontier that none of the alternatives match across all three task types. Two example applications, ovarian cancer biomarker discovery from the PLCO Trial and critical-temperature regression on the UCI Superconductivity Data, illustrate how the framework is used in practice and what trade-offs become visible when stability is treated as a first-class deliverable rather than an emergent property.