Online selective conformal inference: adaptive scores, convergence rate and optimality

2025-08-14

AI Summary

This paper addresses supervised online learning, where predictive uncertainty quantification must simultaneously satisfy long-term average miscoverage control (at level α), instantaneous error rate (IER) constraints at arbitrary time points, and task adaptability. Method: We propose an optional-time online selective conformal prediction framework grounded in conformal prediction, incorporating dynamic calibration scores and online model updating to support diverse selective inference tasks—including extreme-value prediction intervals, rejection-aware classification, and online hypothesis testing. Contribution/Results: We establish non-asymptotic optimality of the method’s convergence rate. Empirically, it precisely controls both IER at user-specified times and long-term average miscoverage, outperforming existing online conformal methods across multiple benchmarks.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

In a supervised online setting, quantifying uncertainty has been proposed in the seminal work of cite{gibbs2021adaptive}. For any given point-prediction algorithm, their method (ACI) produces a conformal prediction set with an average missed coverage getting close to a pre-specified level $α$ for a long time horizon. We introduce an extended version of this algorithm, called OnlineSCI, allowing the user to additionally select times where such an inference should be made. OnlineSCI encompasses several prominent online selective tasks, such as building prediction intervals for extreme outcomes, classification with abstention, and online testing. While OnlineSCI controls the average missed coverage on the selected in an adversarial setting, our theoretical results also show that it controls the instantaneous error rate (IER) at the selected times, up to a non-asymptotical remainder term. Importantly, our theory covers the case where OnlineSCI updates the point-prediction algorithm at each time step, a property which we refer to as {it adaptive} capability. We show that the adaptive versions of OnlineSCI can convergence to an optimal solution and provide an explicit convergence rate in each of the aforementioned application cases, under specific mild conditions. Finally, the favorable behavior of OnlineSCI in practice is illustrated by numerical experiments.

Problem

• Extends ACI for selective online uncertainty quantification
• Controls instantaneous error rate at selected times
• Provides convergence rates for adaptive prediction algorithms

Innovation

• Online selective conformal inference for adaptive coverage
• Controls instantaneous error rate at selected times
• Adaptive updates with optimal convergence rate