Counterfactual Online Conformal Prediction Under Adaptive Logging

2026-09-25

AI Summary

This study addresses the counterfactual coverage bias induced by predictive intervention action selection in adaptive logging. To this end, it proposes an inverse propensity weighted online conformal prediction framework. Methodologically, a debiased calibration mechanism is designed to rectify coverage deficits under rare actions, while a doubly robust variant is introduced to mitigate nuisance bias. Efficient inference is achieved by integrating recursive estimation with online learning algorithms. Theoretically, the proposed framework attains coverage guarantees approaching the information-theoretic lower bound. Empirically, experiments demonstrate that it substantially improves counterfactual coverage and reduces downstream decision regret, all while maintaining compact prediction sets.

Abstract

Online conformal prediction can fail when predictions shape actions and actions determine which outcomes enter calibration. Standard adaptive methods may retain marginal coverage while systematically miscovering the counterfactual outcomes of rarely selected actions. This paper formalizes the failure through counterfactual coverage and introduces Propensity-Weighted Online Conformal Prediction, an inverse-propensity-weighted recursion that debiases calibration. A doubly robust variant further reduces nuisance bias to the product of outcome-model and propensity errors. Under positivity, the resulting coverage rate matches an information-theoretic lower bound up to logarithmic factors. Experiments on synthetic decision tasks, open bandit data, and financial rebalancing show that PW-OCP and DR-OCP improve counterfactual coverage and downstream regret without sacrificing prediction-set sharpness.

Problem

• Online Conformal Prediction
• Counterfactual Coverage
• Adaptive Logging
• Selection Bias

Innovation

• Counterfactual Coverage
• Online Conformal Prediction
• Inverse Propensity Weighting
• Doubly Robust
• Adaptive Logging