Conformalized Interval Arithmetic with Symmetric Calibration

2024-08-20

arXiv.org

AI Summary

Conventional conformal prediction is limited to single-point predictions, failing to quantify uncertainty for aggregate statistics—such as class-wise mean accuracy or total path cost—defined over arbitrary index sets. Method: We propose the first extension of conformal prediction to joint interval estimation of sums or means of unknown labels across arbitrary index sets. Our symmetrically calibrated conformalized interval arithmetic framework ensures exact marginal coverage under permutation invariance, relaxing the single-target constraint without distributional assumptions. Contribution/Results: The method is theoretically rigorous and computationally tractable. Experiments on class-average estimation and path-cost prediction demonstrate that it achieves exact nominal coverage while yielding significantly tighter intervals than state-of-the-art conformal and non-conformal baselines, establishing new performance benchmarks for multi-label uncertainty quantification.

Technology Category

Machine Learning: Calibration & Uncertainty QuantificationReasoning under Uncertainty: Uncertainty RepresentationsConstraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

Security and Privacy: Large-scale security measurementsSearch and Retrieval-Augmented AI: Web evaluation methodologies and metricsUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Uncertainty quantification is essential in decision-making, especially when joint distributions of random variables are involved. While conformal prediction provides distribution-free prediction sets with valid coverage guarantees, it traditionally focuses on single predictions. This paper introduces novel conformal prediction methods for estimating the sum or average of unknown labels over specific index sets. We develop conformal prediction intervals for single target to the prediction interval for sum of multiple targets. Under permutation invariant assumptions, we prove the validity of our proposed method. We also apply our algorithms on class average estimation and path cost prediction tasks, and we show that our method outperforms existing conformalized approaches as well as non-conformal approaches.

Problem

• Extends conformal prediction to sum or average estimation
• Develops valid prediction intervals for multiple target sums
• Outperforms existing methods in class average and cost prediction

Innovation

• Conformal prediction for sum of multiple targets
• Symmetric calibration for joint distribution uncertainty
• Permutation invariant method with valid coverage guarantees