Conformal Prediction Assessment: A Framework for Conditional Coverage Evaluation and Selection

2026-03-28

AI Summary

This work addresses the limitation of traditional conformal prediction, which guarantees marginal coverage but often fails to achieve valid conditional coverage within subpopulations, while existing evaluation methods suffer from the curse of dimensionality. The authors propose a Conformal Prediction Analysis (CPA) framework that reframes conditional coverage assessment as a supervised learning task by training a reliability estimator to predict instance-level coverage probabilities. They introduce a Conditional Validity Index (CVI) to quantify the local safety and efficiency of conformal predictors. Theoretical analysis establishes the convergence of CVI and proves the consistency of CC-Select, a CVI-based model selection algorithm. Empirical results demonstrate that CPA effectively identifies local coverage failures and that CC-Select reliably selects models with superior conditional coverage.

Abstract

Conformal prediction provides rigorous distribution-free finite-sample guarantees for marginal coverage under the assumption of exchangeability, but may exhibit systematic undercoverage or overcoverage for specific subpopulations. Assessing conditional validity is challenging, as standard stratification methods suffer from the curse of dimensionality. We propose Conformal Prediction Assessment (CPA), a framework that reframes the evaluation of conditional coverage as a supervised learning task by training a reliability estimator that predicts instance-level coverage probabilities. Building on this estimator, we introduce the Conditional Validity Index (CVI), which decomposes reliability into safety (undercoverage risk) and efficiency (overcoverage cost). We establish convergence rates for the reliability estimator and prove the consistency of CVI-based model selection. Extensive experiments on synthetic and real-world datasets demonstrate that CPA effectively diagnoses local failure modes and that CC-Select, our CVI-based model selection algorithm, consistently identifies predictors with superior conditional coverage performance.

Problem

• Conformal Prediction
• Conditional Coverage
• Model Selection
• Reliability Estimation
• Subpopulation Validity

Innovation

• Conformal Prediction
• Conditional Coverage
• Reliability Estimator
• Conditional Validity Index
• Model Selection