nonconformity score design

This work addresses the significant inflation of prediction sets in Bayesian-assisted conformal prediction when the prior is misspecified, which undermines predictive efficiency. To mitigate this issue, the authors propose the RoBAS framework, which constructs robust nonconformity scores adaptive to prior quality—enhancing efficiency when the prior is reliable and automatically reverting to a robust baseline when it is not. The approach innovatively introduces two novel score functions grounded in heavy-tailed Bayesian working models and closed-form empirical Bayes shrinkage, effectively integrating Bayesian modeling, conformal prediction, and the Distance-to-Average benchmark. Empirical results demonstrate that RoBAS substantially narrows prediction interval widths under distributional shift while maintaining coverage performance comparable to existing methods in the absence of such shifts.

This work addresses the challenge that conventional Bayesian credible intervals often fail to achieve reliable predictive coverage and exhibit unstable prediction set sizes under model misspecification and distributional shift. The authors propose the first formulation of Bayesian conformal prediction as a decision-theoretic risk minimization problem, introducing the Bayesian posterior predictive density as a nonconformity score and integrating it via Bayesian numerical integration within the split conformal prediction framework to minimize expected prediction set size. Evaluated on both regression and classification tasks—including distribution-shift benchmarks such as ImageNet-A—the method achieves near-nominal 80% coverage (empirically 81%), substantially outperforming traditional Bayesian intervals (49% coverage), while significantly reducing inter-run variability in prediction set size, thereby offering both reliability and stability.

Existing multiple testing methods often suffer from limited statistical power and imprecise false discovery rate (FDR) control due to reliance on only a subset of data—e.g., labeled null or alternative instances. To address this, we propose a unified conformal inference framework that, for the first time, systematically integrates null samples, alternative samples, and unlabeled data to construct nonconformity scores and calibrate p-values via full permutation. Unlike conventional approaches, our framework requires no additional data splitting, automatically selects the optimal conformal method, and is both model-agnostic and computationally efficient. Experiments demonstrate that our method achieves strict FDR control while significantly improving statistical power and cross-scenario robustness. It provides a scalable, robust paradigm for uncertainty quantification in high-dimensional multiple testing, advancing beyond traditional sample-partitioning strategies.

In online learning settings, existing multi-model conformal prediction methods lack theoretical guarantees for model selection and aggregation. Method: This paper proposes the first online conformal model aggregation framework based on temporal weighted voting, integrating conformal prediction, online learning, and an empirical coverage-driven weight update mechanism to enable real-time, adaptive adjustment of model weights. The framework rigorously maintains $1-alpha$ marginal coverage while dynamically optimizing prediction set quality. Contribution/Results: Unlike conventional paradigms requiring a pre-specified single model, our approach supports seamless integration of heterogeneous model streams. Evaluated on multiple data stream benchmarks, it significantly reduces average prediction set width—achieving both statistical reliability (via guaranteed coverage) and practical utility (via tighter, adaptive intervals).

This paper investigates the robustness of split conformal prediction under data contamination: when a small fraction of calibration samples—e.g., due to label noise or distributional shift—originate from a contaminated distribution, standard methods suffer significant degradation in coverage on clean test points. To address this, we propose Contamination Robust Conformal Prediction (CRCP), a novel framework featuring quantile-robust calibration, empirical contamination distribution modeling, and theoretical analysis of coverage error bounds. We theoretically establish that the coverage loss is bounded above by a linear function of the contamination proportion. Empirically, CRCP maintains over 90% nominal coverage even under 10% contamination, substantially outperforming standard conformal prediction, while preserving predictive efficiency and practical applicability.

This work addresses a critical challenge in AI model deployment: how to selectively abstain from predictions under uncertainty while providing rigorous, finite-sample risk control for trusted predictions. The authors propose the SCoRE framework, which introduces—for the first time—a general e-value–based risk control mechanism that integrates conformal inference with hypothesis testing. Requiring only data exchangeability and no modeling assumptions or uniform convergence conditions, SCoRE delivers finite-sample–guaranteed reliable decisions for any pretrained model and any user-specified bounded continuous risk. Notably, the method naturally accommodates distributional shifts. Empirical evaluations across drug discovery, health risk prediction, and large language models demonstrate that SCoRE effectively enforces strict control over positive-class risk.

This work addresses the limitations of existing conformal prediction methods, which typically guarantee only marginal coverage and struggle to ensure conditional coverage for heterogeneous test points or subpopulations, while lacking a unified theoretical framework to analyze their asymptotic validity, compare approaches, or extend them to structured data. The paper proposes the first unified theoretical framework tailored for conditional coverage, deriving non-asymptotic bounds on conditional miscoverage via pointwise and Lₚ paths. It systematically characterizes the sources of error underlying asymptotic conditional validity and provides a coherent interpretation of existing methods. Built upon a weighted symmetry formulation, the framework facilitates conditional coverage–oriented model selection, localization under covariate shift, and natural extensions to structured data. Numerical experiments corroborate the theoretical findings, establishing a comparable, extensible, and practically informative paradigm for conditional coverage.

This study addresses the misalignment between conventional prediction-oriented scoring and selection objectives in complex target regions—such as intervals, variance-driven sets, multimodal distributions, or multi-condition scenarios. To resolve this, the authors propose using target membership probability as a nonconformity score to directly rank binary selection events, combined with a null-calibrated conformal selection (NCCS) procedure that leverages non-target calibration samples to produce finite-sample valid p-values. This approach is the first to explicitly distinguish prediction from selection tasks and formalizes a target membership scoring principle. While maintaining comparable performance under mean-monotonic targets, it substantially improves selection efficacy in variance-driven and other complex settings. Moreover, in rare-target regimes, NCCS effectively mitigates the anti-conservatism of empirical FDP thresholds, achieving both high power and rigorous control of the false discovery rate in finite samples.

This work proposes an interpretable statistical inference framework that decomposes predictive scoring functions into three components—calibration error, discrimination ability, and uncertainty—applicable to multi-step-ahead point forecasts such as means and quantiles, and compatible with both smooth and nonsmooth scoring rules. Building on linear recalibration and integrating Mincer–Zarnowitz regression with asymptotic inference theory, the method delivers the first fully interpretable tripartite decomposition for general scoring functions, unifying and extending classical calibration tests and predictive performance evaluation. Empirical applications to inflation surveys and financial risk models reveal critical discrepancies obscured by aggregate scores, exposing a misalignment between backtesting practices and predictive accuracy in banking regulation, thereby substantially enhancing the informativeness and statistical power of forecast evaluation.

This work addresses the limitation of traditional conformal prediction, which guarantees marginal coverage but often fails to achieve valid conditional coverage within subpopulations, while existing evaluation methods suffer from the curse of dimensionality. The authors propose a Conformal Prediction Analysis (CPA) framework that reframes conditional coverage assessment as a supervised learning task by training a reliability estimator to predict instance-level coverage probabilities. They introduce a Conditional Validity Index (CVI) to quantify the local safety and efficiency of conformal predictors. Theoretical analysis establishes the convergence of CVI and proves the consistency of CC-Select, a CVI-based model selection algorithm. Empirical results demonstrate that CPA effectively identifies local coverage failures and that CC-Select reliably selects models with superior conditional coverage.