Online Conformal Prediction Beyond Feedback

2026-08-07

AI Summary

This work addresses the problem of feedback-free online conformal prediction for black-box classifiers operating on non-i.i.d. data streams in safety-critical settings where predictive feedback is unavailable. At each round, the learner must choose between outputting a prediction set or querying the true label, but not both. The authors innovatively frame this setting as a partial monitoring game and integrate online learning with conformal prediction theory to develop a label-efficient algorithm. Their method achieves an expected coverage no less than $\beta - O(T^{-1/3})$ while querying labels at a rate of only $O(T^{-1/3})$, and attains a regret bound of $O(T^{2/3})$. Empirical evaluations demonstrate the approach’s effectiveness across arbitrary black-box models and non-stationary data streams.

Abstract

Uncertainty quantification is essential when deploying machine learning models in safety-critical applications. Online conformal prediction (OCP) provides theoretically principled uncertainty quantification for arbitrary black-box classifiers and non-i.i.d. data streams by constructing prediction sets that are guaranteed to contain the true label at a user-specified frequency. OCP usually updates prediction sets using feedback from previously deployed predictions. We instead study an OCP setting beyond feedback: on each round, the learner can either output a prediction set or query the correct label, but not both. Thus, no deployed prediction is ever evaluated directly. We reduce this problem to a partial monitoring game in which prediction actions return no observation and a separate query action reveals the label. The reward function is constructed in a way that encourages the learner to output small prediction sets while ensuring that the correct label is covered with a sufficiently high probability. To solve this game, we develop OCP with queries (OCPQ) by adapting the label efficient forecaster of Cesa-Bianchi, Lugosi, and Stoltz (2004) to our setting. For any black box classifier and any (non-i.i.d.) oblivious data stream of length $T$, OCPQ has $O(T^{2/3})$ expected regret and expected coverage at least $β-O(T^{-1/3})$ for a user-defined $β$, while querying only an expected $T^{-1/3}$ fraction of rounds. This provides coverage comparable to bandit-based OCP methods while requiring no feedback from deployed prediction sets. Experiments on real-world datasets further demonstrate the effectiveness of our approach.

Problem

• Online Conformal Prediction
• Uncertainty Quantification
• Partial Monitoring
• Label Querying
• Non-i.i.d. Data Streams

Innovation

• Online Conformal Prediction
• Partial Monitoring
• Label-Efficient Learning
• Uncertainty Quantification
• Non-i.i.d. Data Streams