🤖 AI Summary
Standard boosting methods often suffer from redundancy among base learners due to repeatedly fitting correlated errors. This work proposes SCBoost, a novel framework that reformulates boosting from a geometric perspective. SCBoost introduces Spectral Residual Projection (SRP) to constrain each new learner to the orthogonal complement of the subspace spanned by previous predictions, ensuring it captures only previously unexplained information. Additionally, Covariance-Regularized Weighting (CRW) is employed to optimize ensemble weights, explicitly reducing inter-learner correlation. The approach enables an exact additive decomposition of residual energy and provably enhances the signal-to-noise ratio under isotropic noise assumptions. Empirical evaluations across ten benchmark datasets demonstrate that SCBoost significantly outperforms baseline methods, achieving particularly notable gains in accuracy and F1 score.
📝 Abstract
While sequential residual fitting is the bedrock of standard boosting frameworks, it inherently breeds learner redundancy by repeatedly revisiting correlated error components. To address this bottleneck, we propose a shift from residual fitting to \textit{residual orthogonalization} and introduce SCBoost. Our framework tackles redundancy through two complementary mechanisms: Spectral Residual Projection (SRP) and Covariance-Regularized Weighting (CRW). During training, SRP projects each residual target onto the orthogonal complement of the historical prediction subspace, forcing successive learners to capture only novel empirical innovations. During aggregation, CRW optimizes ensemble weights on a validation set with an explicit covariance penalty to mitigate remaining correlations. Theoretically, we provide a finite-sample geometric characterization proving that SRP yields an exact additive residual-energy decomposition. Furthermore, under an isotropic-noise assumption, we rigorously establish the conditions under which this projection improves the effective Signal-to-Noise Ratio. Extensive experiments across ten benchmark datasets demonstrate that SCBoost delivers strong out-of-the-box performance, particularly in accuracy and F1 score. This work reinterprets boosting through a geometric lens, suggesting that explicit redundancy control is a principled and necessary step toward more efficient ensemble architectures.