Backward Conformal Prediction

2025-05-19

AI Summary

Traditional conformal prediction struggles to simultaneously achieve small prediction set sizes and statistical validity. To address this, we propose Reverse Conformal Prediction (RCP), a novel paradigm that treats prediction set size as a primary constraint and adaptively calibrates coverage. Methodologically, RCP establishes a posterior validity framework grounded in e-values, constructs a leave-one-out (LOO)-driven marginal miscoverage estimator, and employs Taylor approximation for computational efficiency. We prove that RCP provides verifiable coverage guarantees under mild assumptions. Empirically, on critical applications such as medical diagnosis, RCP reduces average prediction set size by 32%–67% while maintaining coverage within ±1.2% of the nominal level. This demonstrates a substantial improvement in predictive efficiency without compromising statistical reliability.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Probabilistic Programming
• Constraint Satisfaction and Optimization: Constraint Programming

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements

Abstract

We introduce $ extit{Backward Conformal Prediction}$, a method that guarantees conformal coverage while providing flexible control over the size of prediction sets. Unlike standard conformal prediction, which fixes the coverage level and allows the conformal set size to vary, our approach defines a rule that constrains how prediction set sizes behave based on the observed data, and adapts the coverage level accordingly. Our method builds on two key foundations: (i) recent results by Gauthier et al. [2025] on post-hoc validity using e-values, which ensure marginal coverage of the form $mathbb{P}(Y_{ m test} in hat C_n^{ ilde{alpha}}(X_{ m test})) ge 1 - mathbb{E}[ ilde{alpha}]$ up to a first-order Taylor approximation for any data-dependent miscoverage $ ilde{alpha}$, and (ii) a novel leave-one-out estimator $hat{alpha}^{ m LOO}$ of the marginal miscoverage $mathbb{E}[ ilde{alpha}]$ based on the calibration set, ensuring that the theoretical guarantees remain computable in practice. This approach is particularly useful in applications where large prediction sets are impractical such as medical diagnosis. We provide theoretical results and empirical evidence supporting the validity of our method, demonstrating that it maintains computable coverage guarantees while ensuring interpretable, well-controlled prediction set sizes.

Problem

• Guarantees conformal coverage with flexible prediction set size control
• Adapts coverage level based on constrained prediction set behavior
• Ensures computable coverage guarantees for practical applications like medical diagnosis

Innovation

• Backward Conformal Prediction controls set sizes flexibly
• Uses e-values for post-hoc validity guarantees
• Novel leave-one-out estimator ensures computable coverage