calibration techniques

In safety-critical applications, evaluating uncertainty calibration of regression models is hindered by inconsistent metric definitions, conflicting assumptions, and incomparable scales—impeding interpretability and reproducibility. This work systematically surveys and categorizes existing calibration metrics, then conducts a model-agnostic benchmark across real-world, synthetic, and manually miscalibrated datasets. We empirically demonstrate—for the first time—that most metrics yield contradictory or even opposing conclusions for identical calibration states, confirming that metric choice critically influences research outcomes. To address this, we propose ENCE (Expected Normalized Calibration Error) and CWC (Weighted Coverage Confidence) as more robust and stable primary metrics. Experiments across diverse scenarios show that ENCE and CWC exhibit superior consistency, strong resilience to noise and distribution shifts, and enhanced interpretability. Our findings establish a reproducible methodological foundation for uncertainty calibration evaluation in regression.

This paper addresses the reliability of calibration evaluation for machine learning models, identifying systematic biases in the widely used Expected Calibration Error (ECE) under distributional shift and varying binning strategies. Methodologically, it clarifies the logical hierarchy among multi-level calibration definitions, and systematically exposes ECE’s limitations through visualization, binning-based statistical analysis, and theoretical derivation—demonstrating its failure to satisfy key requirements of robustness and consistency in calibration assessment. Building on this critique, the paper introduces and explicates emerging calibration paradigms—including distribution-level and instance-level calibration—alongside their corresponding evaluation methodologies, thereby constructing a rigorous, interpretable, and practice-oriented calibration knowledge framework. The results equip researchers with principled guidance for selecting appropriate evaluation metrics and advance calibration assessment from ad hoc, heuristic practices toward standardization and formalization.

Existing studies lack rigorous theoretical characterization of posterior calibration methods—such as Platt scaling and isotonic regression—particularly regarding their dependence on feature quality, generalizability across models and datasets, and convergence behavior and robustness under finite-sample regimes. Method: We establish a unified theoretical framework for these two dominant calibration paradigms, deriving the first non-asymptotic guarantees on convergence rates, computational complexity, and explicit sample-size dependencies. Our analysis quantifies the relationship between feature informativeness and calibration robustness. Results: Through synthetic experiments and extensive empirical evaluation across diverse model architectures and benchmark datasets, we validate our theoretical findings. The results yield actionable guidance: isotonic regression is preferable under low signal-to-noise ratios or limited samples, whereas Platt scaling exhibits superior robustness in high-dimensional sparse feature settings. Our work provides interpretable, reusable principles for uncertainty calibration in practical machine learning systems.

Modern deep neural networks often exhibit miscalibration—exhibiting overconfidence in incorrect predictions and insufficient confidence discrimination among correct ones. Method: This paper proposes a post-hoc calibration method that explicitly leverages prediction correctness as a supervisory signal. It introduces a novel “prediction correctness awareness” mechanism, which implicitly models correctness via lightweight input transformations (e.g., rotation, grayscale conversion), thereby eliminating reliance on non-calibration-oriented losses like cross-entropy. A transformation-augmented calibration objective is designed to enable end-to-end training on unlabeled validation data. Contribution/Results: Extensive experiments demonstrate state-of-the-art calibration performance both in-distribution and out-of-distribution. The method significantly mitigates overconfidence on erroneous predictions while enhancing discriminability of confidence scores for correct predictions, all without requiring ground-truth labels during calibration.

This work addresses the challenge of entangled uncertainty sources and the difficulty of disentangling pointwise statistical risk in predictive modeling. We propose a unified generative framework based on approximate Bayesian inference that, for the first time, establishes an explicit, interpretable decomposition linking pointwise statistical risk to two fundamental uncertainty types: aleatoric uncertainty (arising from inherent data noise) and epistemic uncertainty (stemming from model ignorance). The framework jointly generates multiple uncertainty measures while ensuring semantic consistency across them. Experiments on image benchmarks demonstrate significant improvements in out-of-distribution detection and misclassification identification, achieving higher AUROC scores compared to existing methods. Our approach thus provides robust, quantifiable uncertainty estimates essential for downstream uncertainty-aware tasks such as active learning, safe decision-making, and model debugging.

This work addresses the limitation of existing safety-critical systems, which typically evaluate only predictive accuracy while lacking rigorous validation of the overall calibration of predicted probability distributions. To bridge this gap, the authors propose a modular calibration testing framework that decouples the calibration process into four interchangeable components: data model, scoring rule, hypothesis formulation, and statistical test procedure. Built upon formal statistical hypothesis testing, the framework provides a single accept/reject decision for the entire predictive distribution. Crucially, it rejects only overly confident predictions while tolerating reasonable deviations, thereby balancing practicality with flexibility. Empirical evaluations on weather forecasting and robotic pose estimation tasks demonstrate that the framework effectively supports reliable deployment in safety-critical applications.

Industrial prediction and soft sensing often fail due to field data suffering from bias, latency, or seemingly plausible yet unreliable measurements. This work proposes a large language model (LLM)-guided Measurement Credibility Correction (MCC) method that, for the first time, leverages semantic information from process documentation to construct an external reference—requiring neither numerical correlations, fault labels, nor explicit process equations—for lightweight pre-inference correction. MCC translates document semantics into reference signals compatible with numerical models and integrates them at the front end of the prediction pipeline. Evaluated on multiple real-world industrial tasks, MCC reduces average relative MAE by 30.7% on authentic test data and by 80.3% under controlled contamination, while adding only 0.5–2.0k online parameters and incurring a maximum inference latency of 0.089 ms per step.

Approximate Bayesian inference often underestimates true uncertainty due to posterior credible intervals that are excessively narrow. This work proposes two simulation-based calibration (SBC)-driven methods for recalibrating approximate posteriors, systematically leveraging the SBC framework to adjust the width of posterior uncertainty intervals and achieve marginal calibration. The approach is applicable to complex model structures, including hierarchical models, and demonstrates consistent efficacy across diverse experimental settings by meaningfully widening posterior intervals. As a result, the proposed recalibration substantially enhances the calibration accuracy and reliability of approximate Bayesian inference.

This study addresses the limitation of existing posterior calibration evaluation practices, which predominantly focus on aggregate performance while overlooking robustness across varying operational conditions within datasets. The authors propose the first conditionally stratified evaluation framework, employing preregistered experiments to systematically compare the robustness of temperature scaling (TEMP) and isotonic regression (ISO) under four controlled conditions. The assessment encompasses multiple dimensions—including calibration slope, Brier score, discrimination, and AUROC—and applies Holm’s correction for multiplicity in rigorous hypothesis testing. Results demonstrate that TEMP consistently achieves calibration slopes closer to one and superior, more stable Brier scores across all conditions; differences in discrimination between the two methods are negligible; and AUROC performance varies by condition, revealing that the robustness of calibration methods is highly dependent on both specific operational contexts and the choice of evaluation metric.

This work addresses the challenge that probabilistic programs generated by language models often suffer from statistical misspecifications—such as incorrect likelihoods, priors, or parameterizations—that are difficult to detect with conventional unit tests. The paper introduces, for the first time, Bayesian calibration as a central criterion for assessing the correctness of probabilistic programs and proposes a fully unsupervised, reference-free framework for their detection and repair. By integrating Bayesian validation techniques—including posterior predictive checks, simulation-based calibration (SBC), sampling diagnostics (e.g., $\hat{R}$, divergences, effective sample size), and held-out predictive log density—the method generates feedback signals to drive an iterative repair loop within large language models. Evaluated on 200 instances, the approach achieves detection AUCs of 0.97 with reference programs and 62–78% without, substantially outperforming unit testing; repair success rates reach 92% and 100% using GPT-5.1 and Claude, respectively.