Tight Generalization Bound for AdaBoost

2026-07-29

AI Summary

This work addresses the looseness of existing generalization error bounds for AdaBoost by proposing a novel approach grounded in margin analysis and VC dimension theory. By characterizing the property that AdaBoost’s output classifier incurs zero empirical γ/2-margin loss, the authors derive a tight margin-based generalization bound for voting classifiers. They establish, for the first time, an upper bound on AdaBoost’s generalization error that matches known lower bounds: Θ(d ln(nγ²/d)/(nγ²) + ln(1/δ)/n), where d denotes the VC dimension of the base hypothesis class, γ represents the edge of the weak learner, n is the sample size, and δ is the confidence parameter. This result provides a theoretically rigorous and practically insightful characterization of AdaBoost’s generalization behavior, filling a critical gap in its theoretical understanding.

Abstract

In this paper we show that the generalization error of AdaBoost is $Θ\big(\tfrac{d\ln(nγ^{2}/d)}{nγ^2}+\tfrac{\ln(1/δ)}{n}\big)$, where $γ$ is the advantage guaranteed by the weak learner, $d$ is the VC-dimension of the class containing the weak hypotheses, $n$ is the sample size, and $δ$ is the confidence parameter. The contribution of this paper is the upper bound; the matching lower bound follows from prior work. The upper bound proof follows by combining the known fact that AdaBoost outputs a voting classifier whose voting function has zero empirical $γ/2$-margin loss with what is, to the best of our knowledge, a new margin-based generalization bound for voting classifiers.

Problem

• AdaBoost
• generalization bound
• margin theory
• VC-dimension
• voting classifier

Innovation

• AdaBoost
• generalization bound
• margin theory
• VC-dimension
• voting classifier