🤖 AI Summary
To address the unreliability of prediction confidence in deep neural network image classification, this paper proposes a lightweight meta-model classifier ensemble method that achieves efficient uncertainty calibration without requiring additional calibration data. The approach constructs a parameter-efficient meta-model to fuse outputs from multiple base classifiers and jointly evaluates calibration performance using majority voting alongside Expected Calibration Error (ECE) and Maximum Calibration Error (MCE). For the first time, both theoretical analysis and empirical evaluation demonstrate its significant calibration advantages: across diverse mainstream CNN architectures, it reduces ECE and MCE by over 40% on average while preserving classification accuracy nearly unchanged; moreover, its parameter count is 3–5× smaller than conventional model ensembles. This work establishes a novel paradigm for high-reliability, low-overhead model calibration.
📝 Abstract
This paper investigates novel classifier ensemble techniques for uncertainty calibration applied to various deep neural networks for image classification. We evaluate both accuracy and calibration metrics, focusing on Expected Calibration Error (ECE) and Maximum Calibration Error (MCE). Our work compares different methods for building simple yet efficient classifier ensembles, including majority voting and several metamodel-based approaches. Our evaluation reveals that while state-of-the-art deep neural networks for image classification achieve high accuracy on standard datasets, they frequently suffer from significant calibration errors. Basic ensemble techniques like majority voting provide modest improvements, while metamodel-based ensembles consistently reduce ECE and MCE across all architectures. Notably, the largest of our compared metamodels demonstrate the most substantial calibration improvements, with minimal impact on accuracy. Moreover, classifier ensembles with metamodels outperform traditional model ensembles in calibration performance, while requiring significantly fewer parameters. In comparison to traditional post-hoc calibration methods, our approach removes the need for a separate calibration dataset. These findings underscore the potential of our proposed metamodel-based classifier ensembles as an efficient and effective approach to improving model calibration, thereby contributing to more reliable deep learning systems.