🤖 AI Summary
This paper addresses the instability and curse of dimensionality in density-ratio estimation for high-dimensional causal inference. We propose a direct density-ratio learning method grounded in a unified framework integrating Bregman divergences and the Riesz representation theorem. Unlike conventional two-stage density estimation, our approach formulates the density ratio as a regression problem under the Riesz representation, using a Bregman divergence as the loss function, and incorporates classification-inspired objectives and data augmentation to mitigate estimation bias under unobserved intervention distributions. To our knowledge, this is the first work to theoretically unify Bregman divergences, Riesz regression, and probabilistic classification, while supporting diverse model classes—including gradient boosting, neural networks, and kernel methods. Extensive simulations demonstrate that different Bregman divergences and augmentation strategies enhance robustness. A publicly available Python package enables flexible implementation.
📝 Abstract
The ratio of two probability density functions is a fundamental quantity that appears in many areas of statistics and machine learning, including causal inference, reinforcement learning, covariate shift, outlier detection, independence testing, importance sampling, and diffusion modeling. Naively estimating the numerator and denominator densities separately using, e.g., kernel density estimators, can lead to unstable performance and suffers from the curse of dimensionality as the number of covariates increases. For this reason, several methods have been developed for estimating the density ratio directly based on (a) Bregman divergences or (b) recasting the density ratio as the odds in a probabilistic classification model that predicts whether an observation is sampled from the numerator or denominator distribution. Additionally, the density ratio can be viewed as the Riesz representer of a continuous linear map, making it amenable to estimation via (c) minimization of the so-called Riesz loss, which was developed to learn the Riesz representer in the Riesz regression procedure in causal inference. In this paper we show that all three of these methods can be unified in a common framework, which we call Bregman-Riesz regression. We further show how data augmentation techniques can be used to apply density ratio learning methods to causal problems, where the numerator distribution typically represents an unobserved intervention. We show through simulations how the choice of Bregman divergence and data augmentation strategy can affect the performance of the resulting density ratio learner. A Python package is provided for researchers to apply Bregman-Riesz regression in practice using gradient boosting, neural networks, and kernel methods.