Partition Trees: Conditional Density Estimation over General Outcome Spaces

2026-02-03

AI Summary

This study addresses conditional density estimation in a general outcome space encompassing both continuous and categorical variables, without imposing strong parametric assumptions on the target distribution. To this end, the authors propose a tree-based nonparametric framework that constructs piecewise constant density models through adaptive partitioning and directly minimizes the conditional negative log-likelihood to learn the tree structure. The approach is further extended into an ensemble method termed Partition Forests. This work represents the first integration of nonparametric piecewise constant density modeling with conditional density estimation, enabling unified handling of mixed-type outputs. Experimental results demonstrate that the method outperforms CART-based approaches in probabilistic prediction and achieves performance comparable to or better than state-of-the-art probabilistic trees and random forests, while exhibiting robustness to redundant features and heteroscedastic noise.

Technology Category

• Machine Learning: Ensemble Methods
• Reasoning under Uncertainty: Relational Probabilistic Models
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising

Abstract

We propose Partition Trees, a tree-based framework for conditional density estimation over general outcome spaces, supporting both continuous and categorical variables within a unified formulation. Our approach models conditional distributions as piecewise-constant densities on data adaptive partitions and learns trees by directly minimizing conditional negative log-likelihood. This yields a scalable, nonparametric alternative to existing probabilistic trees that does not make parametric assumptions about the target distribution. We further introduce Partition Forests, an ensemble extension obtained by averaging conditional densities. Empirically, we demonstrate improved probabilistic prediction over CART-style trees and competitive or superior performance compared to state-of-the-art probabilistic tree methods and Random Forests, along with robustness to redundant features and heteroscedastic noise.

Problem

• conditional density estimation
• general outcome spaces
• tree-based methods
• nonparametric modeling
• probabilistic prediction

Innovation

• Partition Trees
• conditional density estimation
• nonparametric
• data-adaptive partitions
• Partition Forests