🤖 AI Summary
Existing classification performance metrics suffer from inconsistent scales and high sensitivity to class imbalance, rendering cross-dataset evaluation incomparable and difficult to interpret. To address this, we propose the Outperformance Score (OS), a unified normalization framework that maps any confusion-matrix-based metric onto the [0,1] interval. OS is defined as the percentile rank of an observed performance value within a reference empirical distribution induced by the class imbalance ratio. This constitutes the first general-purpose standardization framework that endows diverse CMBCP metrics with consistent semantics and comparable scales. Crucially, OS eliminates reliance on fixed decision thresholds or parametric distributional assumptions, enabling robust evaluation under dynamically varying imbalance ratios. Extensive validation across real-world datasets from healthcare, finance, and natural language processing domains demonstrates that OS significantly enhances reliability and consistency in both inter-metric and cross-dataset comparisons, while supporting plug-and-play integration.
📝 Abstract
Many classification performance metrics exist, each suited to a specific application. However, these metrics often differ in scale and can exhibit varying sensitivity to class imbalance rates in the test set. As a result, it is difficult to use the nominal values of these metrics to interpret and evaluate classification performances, especially when imbalance rates vary. To address this problem, we introduce the outperformance score function, a universal standardization method for confusion-matrix-based classification performance (CMBCP) metrics. It maps any given metric to a common scale of $[0,1]$, while providing a clear and consistent interpretation. Specifically, the outperformance score represents the percentile rank of the observed classification performance within a reference distribution of possible performances. This unified framework enables meaningful comparison and monitoring of classification performance across test sets with differing imbalance rates. We illustrate how the outperformance scores can be applied to a variety of commonly used classification performance metrics and demonstrate the robustness of our method through experiments on real-world datasets spanning multiple classification applications.