Calibration of ordinal regression networks

2024-10-21

arXiv.org

AI Summary

Deep neural networks for ordinal regression often suffer from overconfident and poorly calibrated predictions due to cross-entropy loss, while their softmax outputs violate the ordinal unimodality constraint. Existing approaches focus on ordinal modeling but neglect calibration. This paper proposes the Ordinal-Calibrated Loss (OCL), the first loss function to integrate ordinal-aware calibration: it jointly employs soft ordinal encoding and ordinal-aware regularization to explicitly enforce unimodality and reliability of predicted probabilities during optimization. OCL requires no post-hoc calibration and ensures end-to-end calibration and ordinal consistency. Evaluated on four standard benchmarks, OCL achieves state-of-the-art calibration—reducing expected calibration error (ECE) by 28–41%—while maintaining top-tier classification accuracy.

Technology Category

Machine Learning: Calibration & Uncertainty QuantificationSearch and Optimization: Learning to SearchReasoning under Uncertainty: Relational Probabilistic Models

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingGraph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphsUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Recent studies have shown that deep neural networks are not well-calibrated and often produce over-confident predictions. The miscalibration issue primarily stems from using cross-entropy in classifications, which aims to align predicted softmax probabilities with one-hot labels. In ordinal regression tasks, this problem is compounded by an additional challenge: the expectation that softmax probabilities should exhibit unimodal distribution is not met with cross-entropy. The ordinal regression literature has focused on learning orders and overlooked calibration. To address both issues, we propose a novel loss function that introduces ordinal-aware calibration, ensuring that prediction confidence adheres to ordinal relationships between classes. It incorporates soft ordinal encoding and ordinal-aware regularization to enforce both calibration and unimodality. Extensive experiments across four popular ordinal regression benchmarks demonstrate that our approach achieves state-of-the-art calibration without compromising classification accuracy.

Problem

• Calibrates ordinal regression networks effectively.
• Addresses over-confident predictions in deep learning.
• Ensures unimodal distribution in softmax probabilities.

Innovation

• Novel ordinal-aware loss function
• Soft ordinal encoding technique
• Ordinal-aware regularization method