Optimal Parallelization of Boosting

2024-08-29

Neural Information Processing Systems

AI Summary

A long-standing gap exists between theoretical lower bounds and practical performance in parallelizing Boosting algorithms. Method: Leveraging the sample-optimal weak learner assumption, we integrate adaptive reweighting analysis, a parallel gradient boosting framework, and fine-grained communication-computation decomposition. Contribution/Results: We establish the first tight parallel complexity lower bound—from weak to strong learners—under Boosting, and design a parallel Boosting algorithm that matches this bound up to logarithmic factors across the full parameter spectrum. Our analysis precisely characterizes the optimal trade-off between the number of boosting rounds $p$ and the per-round parallel work $t$. This resolves the fundamental problem of achieving near-sample-optimal parallel complexity in Boosting, thereby bridging the critical gap between theory and practice.

Technology Category

• Machine Learning: Ensemble Methods
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Recent works on the parallel complexity of Boosting have established strong lower bounds on the tradeoff between the number of training rounds $p$ and the total parallel work per round $t$. These works have also presented highly non-trivial parallel algorithms that shed light on different regions of this tradeoff. Despite these advancements, a significant gap persists between the theoretical lower bounds and the performance of these algorithms across much of the tradeoff space. In this work, we essentially close this gap by providing both improved lower bounds on the parallel complexity of weak-to-strong learners, and a parallel Boosting algorithm whose performance matches these bounds across the entire $p$ vs.~$t$ compromise spectrum, up to logarithmic factors. Ultimately, this work settles the true parallel complexity of Boosting algorithms that are nearly sample-optimal.

Problem

• Closing the gap between theoretical lower bounds and parallel boosting algorithms
• Establishing improved lower bounds on parallel complexity of weak-to-strong learners
• Developing parallel boosting algorithm matching bounds across p vs t spectrum

Innovation

• Improved lower bounds on parallel complexity
• Parallel Boosting algorithm matching bounds
• Settles true parallel complexity of Boosting