On Imbalanced Regression with Hoeffding Trees

2026-02-25

AI Summary

This work addresses the challenge of class imbalance in online streaming regression by proposing a novel incremental decision tree model built upon the Hoeffding tree framework, which for the first time integrates kernel density estimation (KDE) and hierarchical shrinkage (HS). The authors adapt KDE to the streaming setting through telescoping arguments to enable smooth prediction, and they reformulate HS for online applicability. Experimental results demonstrate that KDE substantially enhances predictive performance during the early stages of data streams, whereas HS yields no significant improvement under streaming conditions. This study offers a new modeling perspective and technical pathway for handling imbalanced data in streaming regression tasks.

Technology Category

Machine Learning: Kernel MethodsData Mining & Knowledge Management: Data Stream MiningReasoning under Uncertainty: Sequential Decision Making

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingUser Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertisingEconomics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Many real-world applications provide a continuous stream of data that is subsequently used by machine learning models to solve regression tasks of interest. Hoeffding trees and their variants have a long-standing tradition due to their effectiveness, either alone or as base models in broader ensembles. At the same time a recent line of work in batch learning has shown that kernel density estimation (KDE) is an effective approach for smoothed predictions in imbalanced regression tasks [Yang et al., 2021]. Moreover, another recent line of work for batch learning, called hierarchical shrinkage (HS) [Agarwal et al., 2022], has introduced a post-hoc regularization method for decision trees that does not alter the structure of the learned tree. Using a telescoping argument we cast KDE to streaming environments and extend the implementation of HS to incremental decision tree models. Armed with these extensions we investigate the performance of decision trees that may enjoy such options in datasets commonly used for regression in online settings. We conclude that KDE is beneficial in the early parts of the stream, while HS hardly, if ever, offers performance benefits. Our code is publicly available at: https://github.com/marinaAlchirch/DSFA_2026.

Problem

• imbalanced regression
• streaming data
• Hoeffding trees
• kernel density estimation
• hierarchical shrinkage

Innovation

• Hoeffding Trees
• Imbalanced Regression
• Kernel Density Estimation
• Hierarchical Shrinkage
• Streaming Learning