A Rectification-Based Approach for Distilling Boosted Trees into Decision Trees

2025-10-21

AI Summary

This paper addresses the poor interpretability of boosted tree models by proposing an error-correction-based decision tree distillation method. Unlike conventional approaches requiring access to original training data, our method constructs a learnable correction module over the residuals of the boosted tree’s predictions and jointly optimizes both the decision tree structure and leaf-node prediction values, enabling efficient knowledge transfer from the ensemble to a single interpretable tree. Compared to baseline distillation methods such as direct retraining, our approach achieves significantly higher predictive accuracy—averaging a 3.2% AUC improvement across multiple benchmark datasets—while preserving full model interpretability. Key contributions include: (1) a differentiable residual correction mechanism that retains the nonlinear modeling capacity of complex ensembles; and (2) an end-to-end framework for joint tree architecture search and parameter optimization. Experiments demonstrate superior trade-offs between accuracy and interpretability.

Technology Category

Machine Learning: Ensemble MethodsSearch and Optimization: Learning to SearchComputer Vision: Interpretability, Explainability, and Transparency

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingSemantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsUser Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

We present a new approach for distilling boosted trees into decision trees, in the objective of generating an ML model offering an acceptable compromise in terms of predictive performance and interpretability. We explain how the correction approach called rectification can be used to implement such a distillation process. We show empirically that this approach provides interesting results, in comparison with an approach to distillation achieved by retraining the model.

Problem

• Distilling boosted trees into interpretable decision trees
• Balancing predictive performance with model interpretability
• Implementing rectification for improved distillation over retraining

Innovation

• Rectification method distills boosted trees
• Generates interpretable decision tree models
• Improves predictive performance via correction approach