Localized Conformal Prediction for Image Classification with Vision-Language Models

2026-06-30

AI Summary

This work addresses the insufficient quantification of uncertainty in vision-language models for image classification by proposing a novel approach based on locally conformal prediction. The method leverages cosine similarity between test and calibration samples and incorporates a simple yet effective nonlinear transformation strategy to dynamically adjust the size of prediction sets. While rigorously maintaining marginal coverage guarantees, the proposed technique substantially reduces the average prediction set size, achieving statistically significant improvements in natural image classification tasks. This advancement enhances both the precision and practical utility of predictive outputs without compromising reliability.

Abstract

Conformal predictions have attracted significant attention in the field of uncertainty quantification, mainly because of their strong marginal coverage guarantees. Full conditional guarantee is not an attainable goal, a well known fact in conformal predictions literature. As a result, several approaches have tried to approximate this behavior by adapting the conformal sets of test-time samples according to their similarity to calibration examples. Although the latter has gained traction and shown impressive performances for regression problems, its application to image classification remains under-explored. We conduct an extensive benchmarking on natural image classification tasks with vision-language models (VLMs), using our open source implementation of a recent localized conformal prediction algorithm. We show that straightforward usage of the cosine similarity between test-time and calibration visual features, an intuitive choice for VLMs, is not sufficient to improve over the non-local baselines. In response, we propose a simple non-linear transformation of the cosine similarities, which conserves marginal coverage guarantees and achieves statistically significant mean set sizes reduction. Code is available at https://github.com/cfuchs2023/lcp-vlm/.

Problem

• localized conformal prediction
• image classification
• vision-language models
• uncertainty quantification
• calibration

Innovation

• Localized Conformal Prediction
• Vision-Language Models
• Uncertainty Quantification
• Cosine Similarity Transformation
• Image Classification