🤖 AI Summary
This paper addresses computational redundancy and diminishing returns arising from ensemble size expansion in data stream environments. It introduces, for the first time, a linear independence perspective on classifier voting to model ensemble performance. We establish linear independence as a fundamental mechanism for enhancing representational capacity and diversity, and derive a theoretical trade-off framework linking ensemble size to accuracy—yielding the minimal theoretical size required to achieve a target independence probability. Leveraging geometric modeling and weighted majority voting theory, we validate the framework empirically using OzaBagging and GOOWE. Experiments demonstrate that the framework accurately identifies performance saturation points for robust ensembles (e.g., OzaBagging), while revealing how high theoretical diversity may induce decision instability in less robust methods (e.g., GOOWE). The results provide a principled foundation for dynamic ensemble pruning and adaptive size control in streaming settings.
📝 Abstract
Ensemble learning improves classification performance by combining multiple base classifiers. While increasing the number of classifiers generally enhances accuracy, excessively large ensembles can lead to computational inefficiency and diminishing returns. This paper investigates the relationship between ensemble size and performance through the lens of linear independence among classifier votes in data streams. We propose that ensembles composed of linearly independent classifiers maximize representational capacity, particularly under a geometric model. We then generalize the importance of linear independence to the weighted majority voting problem. By modeling the probability of achieving linear independence among classifier outputs, we derive a theoretical framework that explains the trade-off between ensemble size and accuracy. Our analysis leads to a theoretical estimate of the ensemble size required to achieve a user-specified probability of linear independence. We validate our theory through experiments on both real-world and synthetic datasets using two ensemble methods, OzaBagging and GOOWE. Our results confirm that this theoretical estimate effectively identifies the point of performance saturation for robust ensembles like OzaBagging. Conversely, for complex weighting schemes like GOOWE, our framework reveals that high theoretical diversity can trigger algorithmic instability. Our implementation is publicly available to support reproducibility and future research.