Conformal Prediction: A Data Perspective

2024-10-09

arXiv.org

AI Summary

Traditional conformal prediction (CP) struggles with the structural complexity and dynamic nature of multimodal, streaming, and large-scale data. To address this, we reconceptualize CP from a data-centric perspective, proposing a novel methodological framework tailored to modern data science. Our approach designs calibration and ensemble construction strategies adaptable to structured, unstructured, and dynamically evolving data, integrating permutation tests, quantile regression, online learning, and adaptive reweighting—enabling distribution-free uncertainty quantification even for black-box models. It uniformly supports diverse modalities—including images, text, and time series—and introduces a new evaluation criterion that jointly optimizes validity and computational efficiency in large-model and big-data settings. Empirically, our framework significantly enhances CP’s applicability, scalability, and practical utility in real-world complex scenarios.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Conformal prediction (CP), a distribution-free uncertainty quantification (UQ) framework, reliably provides valid predictive inference for black-box models. CP constructs prediction sets that contain the true output with a specified probability. However, modern data science diverse modalities, along with increasing data and model complexity, challenge traditional CP methods. These developments have spurred novel approaches to address evolving scenarios. This survey reviews the foundational concepts of CP and recent advancements from a data-centric perspective, including applications to structured, unstructured, and dynamic data. We also discuss the challenges and opportunities CP faces in large-scale data and models.

Problem

• Addresses uncertainty quantification in black-box models
• Adapts conformal prediction to diverse data modalities
• Explores challenges in large-scale data and model complexity

Innovation

• Conformal prediction ensures valid predictive inference.
• CP constructs probability-based prediction sets.
• Advances address structured, unstructured, dynamic data.