🤖 AI Summary
Standard Bagging ensembles often suffer from overconfidence and redundancy due to uniform voting weights that ignore the varying local competencies of base learners. This work proposes the SCSB framework, which unifies ensemble pruning and probability calibration into a joint optimization problem over the probability simplex. By minimizing out-of-bag loss with an added concave quadratic sparsity-inducing penalty, SCSB overcomes the theoretical limitation of the L1 norm—which fails to induce sparsity on the simplex—while preserving model-agnosticism. The method achieves compression rates up to 96%, substantially reduces expected calibration error, yields linear inference speedup, and maintains or even improves generalization accuracy in most cases.
📝 Abstract
We present Simplex-Constrained Sparse Bagging (SCSB), a mathematically rigorous framework for post-training compression and probability calibration of bootstrap-based bagging ensembles. Standard bagging ensembles (such as Random Forests, Bagged SVMs, and Bagged Neural Networks) assign uniform voting power to all constituent estimators. However, this naive uniform prior ignores the varying local competence of base estimators and contributes to model overconfidence. We formulate ensemble pruning and calibration as a joint optimization problem over the probability simplex by minimizing the Out-Of-Bag (OOB) loss. To induce sparsity, we address the theoretical "L1-simplex paradox" -- the mathematical reality that the L1 norm is constant on the simplex and fails to prune -- by introducing a concave quadratic penalty. SCSB is model-agnostic and achieves up to 96% ensemble compression, yielding linear inference speedups and superior probability calibration (lowered Expected Calibration Error) while preserving or enhancing generalization accuracy.