Conformal Bayes under Label Shift: Post-Hoc Calibration vs. In-Training Adaptation

2026-06-10

AI Summary

This work addresses the challenge of ensuring both valid coverage and geometric efficiency of Conformal Bayes prediction sets under label shift. The authors propose two complementary strategies: posterior calibration, which adjusts the predictive distribution and conformity thresholds via importance weighting, and in-training adaptation, which directly modifies the Bayesian posterior over model parameters to align with the target distribution. For the first time, the study systematically compares these approaches under label shift, revealing that both guarantee valid coverage when training is unbiased; however, under optimization-dominated training, in-training adaptation not only preserves coverage validity but also substantially reduces prediction set width, demonstrating an intrinsic debiasing effect. By integrating Bayesian posterior predictive inference, importance-weighted conformal calibration, and highest posterior density region construction, the method establishes a new paradigm for reliable uncertainty quantification under label shift.

Abstract

Conformal Bayes combines Bayesian posterior predictives with conformal calibration to produce prediction sets that are both statistically valid and geometrically efficient. We study conformal Bayes under label shift from a unified perspective, identifying two complementary approaches that restore nominal target-domain coverage through importance-weighted conformal calibration but operate through independent mechanisms. \emph{Post-hoc calibration} tilts the posterior predictive toward the target domain and corrects the conformal threshold via an importance-weighted quantile, leaving the parameter posterior unchanged. \emph{In-training adaptation} tilts the parameter posterior itself to the target domain, producing a corrected predictive whose highest predictive density region serves as the highest predictive density (HPD) based prediction set under the fitted target predictive; efficiency is model-dependent and does not imply finite-sample conditional optimality. Two controlled experiments show that in an unbiased training regime both strategies achieve valid coverage equally, while in a lead-optimization regime in-training adaptation acts as a debiasing operator, reducing interval width at unchanged coverage.

Problem

• conformal Bayes
• label shift
• post-hoc calibration
• in-training adaptation
• prediction sets

Innovation

• Conformal Bayes
• Label Shift
• Post-Hoc Calibration
• In-Training Adaptation
• Importance Weighting