A calibration test for evaluating set-based epistemic uncertainty representations

2025-02-22

AI Summary

This paper addresses distributional calibration assessment for credal-set-based epistemic uncertainty modeling. We propose the first instance-dependent convex combination calibration test to verify whether a given credal set contains the true data-generating distribution. Our method reformulates calibration testing as an existence problem: determining whether there exists an instance-specific convex combination of credal-set elements under which predictions are distributionally calibrated. To solve this, we integrate differentiable kernel density estimation with proper scoring rule optimization, enabling nonparametric, high-resolution local calibration evaluation. Unlike conventional global calibration tests, our approach exhibits markedly higher sensitivity to local calibration heterogeneity. Empirical validation on both synthetic and real-world datasets confirms its effectiveness. The framework provides a statistically rigorous foundation for assessing the reliability of uncertainty sets in predictive modeling.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

The accurate representation of epistemic uncertainty is a challenging yet essential task in machine learning. A widely used representation corresponds to convex sets of probabilistic predictors, also known as credal sets. One popular way of constructing these credal sets is via ensembling or specialized supervised learning methods, where the epistemic uncertainty can be quantified through measures such as the set size or the disagreement among members. In principle, these sets should contain the true data-generating distribution. As a necessary condition for this validity, we adopt the strongest notion of calibration as a proxy. Concretely, we propose a novel statistical test to determine whether there is a convex combination of the set's predictions that is calibrated in distribution. In contrast to previous methods, our framework allows the convex combination to be instance dependent, recognizing that different ensemble members may be better calibrated in different regions of the input space. Moreover, we learn this combination via proper scoring rules, which inherently optimize for calibration. Building on differentiable, kernel-based estimators of calibration errors, we introduce a nonparametric testing procedure and demonstrate the benefits of capturing instance-level variability on of synthetic and real-world experiments.

Problem

• Calibration test for epistemic uncertainty
• Evaluation of convex credal sets
• Instance-dependent calibration combination

Innovation

• Credal sets for uncertainty representation
• Instance-dependent convex combination learning
• Nonparametric calibration error estimation