Instance-Wise Monotonic Calibration by Constrained Transformation

2025-07-08

AI Summary

Deep neural networks often produce miscalibrated, overconfident probability estimates; existing post-hoc calibration methods struggle to simultaneously preserve instance-level monotonicity—i.e., the original ranking of class probabilities—and expressive power. This paper proposes a novel class of constraint-based, instance-level monotonic calibration methods. We introduce the first linearly parameterized monotonic calibration mapping and formulate calibration as a convex-order-constrained optimization problem, thereby guaranteeing strict monotonicity. Our approach enhances expressivity, interpretability, and robustness without modifying model architecture or requiring retraining, and is applicable to multi-class settings. Extensive experiments across diverse datasets and state-of-the-art models demonstrate that our method consistently outperforms existing SOTA calibration techniques, achieving over 30% average improvement in Expected Calibration Error (ECE) and other calibration metrics. It further exhibits high data efficiency and low computational overhead.

Technology Category

Machine Learning: Calibration & Uncertainty QuantificationKnowledge Representation and Reasoning: Nonmonotonic ReasoningComputer Vision: Adversarial Attacks & Robustness

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and rankingGraph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Deep neural networks often produce miscalibrated probability estimates, leading to overconfident predictions. A common approach for calibration is fitting a post-hoc calibration map on unseen validation data that transforms predicted probabilities. A key desirable property of the calibration map is instance-wise monotonicity (i.e., preserving the ranking of probability outputs). However, most existing post-hoc calibration methods do not guarantee monotonicity. Previous monotonic approaches either use an under-parameterized calibration map with limited expressive ability or rely on black-box neural networks, which lack interpretability and robustness. In this paper, we propose a family of novel monotonic post-hoc calibration methods, which employs a constrained calibration map parameterized linearly with respect to the number of classes. Our proposed approach ensures expressiveness, robustness, and interpretability while preserving the relative ordering of the probability output by formulating the proposed calibration map as a constrained optimization problem. Our proposed methods achieve state-of-the-art performance across datasets with different deep neural network models, outperforming existing calibration methods while being data and computation-efficient. Our code is available at https://github.com/YunruiZhang/Calibration-by-Constrained-Transformation

Problem

• Ensures instance-wise monotonicity in probability calibration
• Improves expressiveness and interpretability of calibration maps
• Achieves robust performance across diverse datasets efficiently

Innovation

• Constrained linear calibration map for monotonicity
• Ensures expressiveness, robustness, and interpretability
• Formulates calibration as constrained optimization problem