Semi-Supervised Conformal Prediction With Unlabeled Nonconformity Score

2025-05-27

AI Summary

Traditional conformal prediction (CP) suffers from poor coverage calibration and excessively wide prediction sets under label scarcity. Method: This paper introduces the first semi-supervised extension of CP, proposing a novel nonconformity measure—Nearest-Neighbor Matching (NNM)—that leverages unlabeled data and pseudo-label similarity to dynamically select reference samples. The approach achieves asymptotically valid marginal and conditional coverage even in low-data regimes, without modifying the underlying predictive model. Contribution/Results: Theoretically, we establish convergence guarantees for coverage validity. Empirically, our method significantly improves coverage stability and prediction set compactness across diverse tasks, while demonstrating robustness to distribution shift. It is compatible with mainstream CP frameworks—including split CP and full CP—and supports both marginal and conditional coverage guarantees.

Technology Category

Machine Learning: Calibration & Uncertainty QuantificationReasoning under Uncertainty: Relational Probabilistic ModelsConstraint Satisfaction and Optimization: Constraint Programming

Application Category

User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systemsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingSecurity and Privacy: Large-scale security measurements

Abstract

Conformal prediction (CP) is a powerful framework for uncertainty quantification, providing prediction sets with coverage guarantees when calibrated on sufficient labeled data. However, in real-world applications where labeled data is often limited, standard CP can lead to coverage deviation and output overly large prediction sets. In this paper, we extend CP to the semi-supervised setting and propose SemiCP, leveraging both labeled data and unlabeled data for calibration. Specifically, we introduce a novel nonconformity score function, NNM, designed for unlabeled data. This function selects labeled data with similar pseudo-label scores to estimate nonconformity scores, integrating them into the calibration process to overcome sample size limitations. We theoretically demonstrate that, under mild assumptions, SemiCP provide asymptotically coverage guarantee for prediction sets. Extensive experiments further validate that our approach effectively reduces instability and inefficiency under limited calibration data, can be adapted to conditional coverage settings, and integrates seamlessly with existing CP methods.

Problem

• Extends conformal prediction to semi-supervised settings
• Addresses coverage deviation with limited labeled data
• Proposes novel nonconformity score for unlabeled data

Innovation

• SemiCP leverages labeled and unlabeled data
• NNM nonconformity score for unlabeled data
• Asymptotic coverage guarantee under mild assumptions