Seeing the Forest for the Trees: The Gaussian Process Limit of BART

2026-07-30

AI Summary

Bayesian Additive Regression Trees (BART) excel in prediction and causal inference but exhibit sensitivity to covariate dimensionality. This work establishes that BART converges, as the number of trees tends to infinity, to a Gaussian process with a specific kernel. Leveraging reproducing kernel Hilbert space (RKHS) theory, the authors propose a random tree feature approximation method that reduces the dependence of the learning rate on dimensionality from polynomial to logarithmic, achieving minimax-optimal convergence rates. The approach substantially enhances computational efficiency and seamlessly integrates with any linear predictor, thereby improving model compatibility without sacrificing theoretical guarantees.

Abstract

Bayesian Additive Regression Trees (BART) have shown state-of-the-art performance in both prediction and causal inference problems. Previous theoretical work has attempted to explain BART's superior performance by establishing posterior contraction rates for standard BART models, but these rates depend strongly on the number of covariates. Here, we take a different approach and study the behavior of BART as the number of trees grows towards infinity. We show that in this regime, BART converges to a Gaussian process (GP) with a particular kernel. The kernel and its corresponding reproducing kernel Hilbert space (RKHS) have favorable inferential properties that help explain BART's excellent performance. We introduce *random tree features* as an approximation to this limiting GP, and establish minimax-optimal learning rates for ridge regression on these random features that depend only logarithmically on dimension. In addition to providing insight into the empirical success of BART, random tree features offer a computational benefit over traditional MCMC estimation. The random-features approximation also allows practitioners to easily incorporate BART into any model which has a linear predictor, expanding the applicability and flexibility of BART.

Problem

• Bayesian Additive Regression Trees
• Gaussian process
• posterior contraction
• high-dimensional inference
• reproducing kernel Hilbert space

Innovation

• Gaussian process
• Bayesian Additive Regression Trees
• random tree features
• minimax-optimal rates
• reproducing kernel Hilbert space