🤖 AI Summary
The randomness in the number of distinct observations within bootstrap resamples—termed sample diversity—contributes to the variance of out-of-bag (OOB) error estimation, yet its isolated impact remains poorly quantified.
Method: To disentangle this effect, we introduce Sequential Bootstrap into the OOB analysis framework, enabling precise, explicit control over the number of unique observations per resample—thereby decoupling diversity variability from other sources of stochasticity. We evaluate it under Breiman’s classic five-OBB experimental design on both synthetic and real-world datasets, using multiple random seeds.
Contribution/Results: Our approach significantly reduces OOB estimation variance without compromising model predictive accuracy, empirically confirming that sample diversity fluctuations constitute a primary source of OOB variance. This work provides the first reproducible, controllable empirical pathway for systematic variance decomposition in bootstrap-based ensembles, thereby advancing both the theoretical understanding and practical toolkit for uncertainty quantification in ensemble learning.
📝 Abstract
Bootstrap resampling is the foundation of many ensemble learning methods, and out-of-bag (OOB) error estimation is the most widely used internal measure of generalization performance. In the standard multinomial bootstrap, the number of distinct observations in each resample is random. Although this source of variability exists, it has rarely been studied in isolation to understand how much it affects OOB-based quantities. To address this gap, we investigate Sequential Bootstrap, a resampling method that forces every bootstrap replicate to contain the same number of distinct observations, and treat it as a controlled modification of the classical bootstrap within the OOB framework. We reproduce Breiman's five original OOB experiments on both synthetic and real-world datasets, repeating all analyses across many different random seeds. Our results show that switching from the classical bootstrap to Sequential Bootstrap leaves accuracy-related metrics essentially unchanged, but yields measurable and data-dependent reductions in several variance-related measures. Therefore, Sequential Bootstrap should not be viewed as a new method for improving predictive performance, but rather as a tool for understanding how randomness in the number of distinct samples contributes to the variance of OOB estimators. This work provides a reproducible setting for studying the statistical properties of resampling-based ensemble estimators and offers empirical evidence that may support future theoretical work on variance decomposition in bootstrap-based systems.