🤖 AI Summary
Probabilistic outputs of AI models often exhibit miscalibration—i.e., predicted confidence scores poorly reflect true accuracy—hindering their reliable deployment in safety-critical applications and ensemble systems. Method: This paper presents a systematic survey of probabilistic calibration evaluation methods for classification and object detection models. Grounded in statistical assessment theory, it unifies diverse approaches—including reliability diagrams, Brier score, expected calibration error (ECE), maximum calibration error (MCE), Kolmogorov–Smirnov test, ROC-based metrics, and IoU-aware measures—within a coherent framework covering binary, multiclass, and detection tasks. Contribution: We propose the first taxonomy of calibration metrics, categorizing 82 existing measures into four families: point-wise, binning-based, kernel/curve-based, and cumulative. Additionally, we introduce the first structured calibration metric knowledge base, enabling rapid metric selection, implementation, and comparative analysis—thereby establishing new interpretable and quantifiable benchmarks for trustworthy AI.
📝 Abstract
Probabilities or confidence values produced by artificial intelligence (AI) and machine learning (ML) models often do not reflect their true accuracy, with some models being under or over confident in their predictions. For example, if a model is 80% sure of an outcome, is it correct 80% of the time? Probability calibration metrics measure the discrepancy between confidence and accuracy, providing an independent assessment of model calibration performance that complements traditional accuracy metrics. Understanding calibration is important when the outputs of multiple systems are combined, for assurance in safety or business-critical contexts, and for building user trust in models. This paper provides a comprehensive review of probability calibration metrics for classifier and object detection models, organising them according to a number of different categorisations to highlight their relationships. We identify 82 major metrics, which can be grouped into four classifier families (point-based, bin-based, kernel or curve-based, and cumulative) and an object detection family. For each metric, we provide equations where available, facilitating implementation and comparison by future researchers.