🤖 AI Summary
Counterfactual prediction under evolving intervention policies or hypothetical decision scenarios remains challenging due to unobservable potential outcomes, hindering model identifiability, evaluation, and generalization. Method: We propose the first systematic theoretical framework addressing this challenge—comprising (i) identifiability conditions for counterfactual prediction models, (ii) a performance evaluation system targeting loss, AUC, and calibration, and (iii) robust hyperparameter selection under model misspecification. Our approach integrates causal inference principles, doubly robust estimation, and loss-driven evaluation metric design. Contribution/Results: Validated via simulation studies and a real-world clinical application—cardiovascular risk prediction in statin-naïve populations—the framework significantly improves out-of-distribution generalization and clinical decision reliability in counterfactual settings.
📝 Abstract
Counterfactual prediction methods are required when a model will be deployed in a setting where treatment policies differ from the setting where the model was developed, or when a model provides predictions under hypothetical interventions to support decision-making. However, estimating and evaluating counterfactual prediction models is challenging because, unlike traditional (factual) prediction, one does not observe the full set of potential outcomes for all individuals. Here, we discuss how to fit or tailor a model to target a counterfactual estimand, how to assess the model's performance, and how to perform model and tuning parameter selection. We provide identifiability and estimation results for building a counterfactual prediction model and for multiple measures of counterfactual model performance including loss-based measures, the area under the receiver operating characteristics curve, and calibration. Importantly, our results allow valid estimates of model performance under counterfactual intervention even if the candidate model is misspecified, permitting a wider array of use cases. We illustrate these methods using simulation and apply them to the task of developing a statin-naive risk prediction model for cardiovascular disease.