Optimal Classification Trees for Continuous Feature Data Using Dynamic Programming with Branch-and-Bound

2025-01-14

AI Summary

Constructing provably optimal classification trees with depth greater than three over continuous features remains computationally intractable due to performance degradation from coarse-grained discretization and artificial depth constraints in existing methods. Method: We propose the first optimization framework that directly searches for globally optimal tree structures over raw continuous data—bypassing discretization entirely. Our approach integrates dynamic programming with branch-and-bound, augmented by a novel similarity-based split pruning strategy and an efficient subroutine for computing optimal-depth binary subtrees. Contribution/Results: Experiments demonstrate that our method achieves 10–100× speedup over state-of-the-art exact algorithms while improving test accuracy by 5% relative to classical greedy heuristics. This work significantly advances the frontier of provably optimal decision tree learning, enhancing both theoretical guarantees and practical scalability.

Technology Category

Search and Optimization: Mixed Discrete/Continuous SearchConstraint Satisfaction and Optimization: Mixed Discrete/Continuous OptimizationMachine Learning: Optimization

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingGraph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphsWeb Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Computing an optimal classification tree that provably maximizes training performance within a given size limit, is NP-hard, and in practice, most state-of-the-art methods do not scale beyond computing optimal trees of depth three. Therefore, most methods rely on a coarse binarization of continuous features to maintain scalability. We propose a novel algorithm that optimizes trees directly on the continuous feature data using dynamic programming with branch-and-bound. We develop new pruning techniques that eliminate many sub-optimal splits in the search when similar to previously computed splits and we provide an efficient subroutine for computing optimal depth-two trees. Our experiments demonstrate that these techniques improve runtime by one or more orders of magnitude over state-of-the-art optimal methods and improve test accuracy by 5% over greedy heuristics.

Problem

• Optimal Classification Tree
• Depth-limited Construction
• Continuous Data

Innovation

• Optimal Classification Tree
• Continuous Data
• Efficiency Improvement