Accuracy-Preserving Calibration via Statistical Modeling on Probability Simplex

2024-02-21

International Conference on Artificial Intelligence and Statistics

AI Summary

Deep neural network (DNN) classifiers require calibration to ensure the reliability of predictive confidence, yet existing probability-simplex-based calibration methods often degrade classification accuracy. To address this, we propose a novel calibration framework grounded in the Concrete distribution for modeling probability simplices. We theoretically establish, for the first time, that DNNs trained with cross-entropy loss are naturally suited as parameterized models within this framework. Furthermore, we design an efficient simplex-aware synthetic sampling training strategy that obviates fine-tuning and thereby preserves original accuracy. Our method enables statistically sound inference over predictive distributions without sacrificing classification performance. Evaluated on standard calibration benchmarks—including ImageNet and CIFAR-100—our approach significantly reduces expected calibration error (ECE) and achieves state-of-the-art calibration performance with zero accuracy loss.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Adversarial Attacks & Robustness
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Classification models based on deep neural networks (DNNs) must be calibrated to measure the reliability of predictions. Some recent calibration methods have employed a probabilistic model on the probability simplex. However, these calibration methods cannot preserve the accuracy of pre-trained models, even those with a high classification accuracy. We propose an accuracy-preserving calibration method using the Concrete distribution as the probabilistic model on the probability simplex. We theoretically prove that a DNN model trained on cross-entropy loss has optimality as the parameter of the Concrete distribution. We also propose an efficient method that synthetically generates samples for training probabilistic models on the probability simplex. We demonstrate that the proposed method can outperform previous methods in accuracy-preserving calibration tasks using benchmarks.

Problem

• Ensures calibration preserves DNN model accuracy.
• Uses Concrete distribution for probability simplex modeling.
• Proposes efficient sample generation for training models.

Innovation

• Uses Concrete distribution for accuracy-preserving calibration
• Theoretically proves DNN optimality with cross-entropy loss
• Efficient synthetic sample generation for training