Analytical and Bootstrap Confidence Intervals of Double Machine Learning: Simulation studies and an application to rural-urban difference in obesity prevalence

2026-07-31

AI Summary

This study addresses the unclear impact of machine learning algorithm choice on confidence interval coverage in double/debiased machine learning (DML), which undermines inference reliability. Through simulations and an empirical analysis of county-level obesity rates in the United States, the authors systematically compare the performance of OLS, LASSO, random forests, LightGBM, and neural networks under both analytical and bootstrap inference procedures. The findings reveal that the choice of learner substantially affects DML coverage accuracy, with a counterintuitive decline in coverage probability as sample size increases. Moreover, higher degrees of rurality are associated with significantly elevated county-level obesity rates. These results underscore the critical role of algorithm selection in ensuring reliable DML inference and offer practical guidance for empirical applications.

Abstract

Double Machine Learning (DML) is a popular approach for treatment effect estimation in various settings, which allows a wide range of flexible machine learning methods to be used for nuisance parameter estimation while preserving valid inference. In practice, however, applied researchers must choose among many machine learning algorithms for nuisance models, and the impact of this choice on the variance estimation of DML is not well characterized. We conduct a comprehensive simulation study to compare the coverage probability of DML confidence intervals across different machine learning algorithms. In this study, we compare (1) analytical confidence intervals derived by DML theory versus (2) bootstrap confidence interval. We use a set of learners including ordinary least squares, LASSO, Random Forest, LightGBM, and Neural Networks under different data generation settings. We evaluate the performance across difference settings by bias, confidence interval width, and most importantly, coverage probability. Our results show substantial variability in coverage performance across analytical and bootstrap confidence intervals, highlighting that learner choice plays a critical role in reliable DML inference. Surprisingly, we find that in many settings, when sample size increases, the coverage probability of both DML analytical and bootstrap confidence interval decreases. We further investigate coverage probabilities using a real dataset on rural urban differences among U.S. counties. The real data analysis discovers that (1) the model performance still varies by the learner choices and (2) greater rurality has a statistically significant increasing effect on county level obesity prevalence.

Problem

• Double Machine Learning
• confidence interval
• coverage probability
• machine learning algorithms
• treatment effect estimation

Innovation

• Double Machine Learning
• Confidence Interval Coverage
• Nuisance Parameter Estimation
• Bootstrap Inference
• Machine Learning Algorithms