Online Selective Conformal Prediction: Errors and Solutions

2025-03-21

AI Summary

In online selective conformal inference, sequentially arriving data are dynamically selected to construct prediction intervals; however, this selection violates exchangeability between test points and the calibration set, leading to distorted selective conditional coverage (SCC) and uncontrolled false coverage rate (FCR). This work first systematically identifies the fundamental theoretical limitations of existing calibration strategies in guaranteeing both SCC and FCR. We propose the first provably exchangeability-restoring randomized calibration set construction method, integrating sequential analysis with conditional independence modeling. We rigorously prove that our approach simultaneously achieves valid SCC guarantees and FCR control under mild assumptions. Empirical evaluation demonstrates substantial improvements in the accuracy–coverage trade-off across diverse online selective settings. Our framework establishes a new statistically reliable paradigm for conformal prediction in dynamic, selection-driven environments.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Sequential Decision Making
• Constraint Satisfaction and Optimization: Other Foundations of Constraint Satisfaction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

In online selective conformal inference, data arrives sequentially, and prediction intervals are constructed only when an online selection rule is met. Since online selections may break the exchangeability between the selected test datum and the rest of the data, one must correct for this by suitably selecting the calibration data. In this paper, we evaluate existing calibration selection strategies and pinpoint some fundamental errors in the associated claims that guarantee selection-conditional coverage and control of the false coverage rate (FCR). To address these shortcomings, we propose novel calibration selection strategies that provably preserve the exchangeability of the calibration data and the selected test datum. Consequently, we demonstrate that online selective conformal inference with these strategies guarantees both selection-conditional coverage and FCR control. Our theoretical findings are supported by experimental evidence examining tradeoffs between valid methods.

Problem

• Address errors in online selective conformal prediction coverage claims
• Propose new calibration strategies to preserve data exchangeability
• Ensure selection-conditional coverage and false coverage rate control

Innovation

• Online selective conformal prediction intervals
• Novel calibration selection strategies
• Guarantees selection-conditional coverage