🤖 AI Summary
Existing binary classifier evaluation lacks a consensus standard, and common metrics are frequently misapplied in class-imbalanced and cost-sensitive settings. Method: We propose Weighted Accuracy (WA) as a unified evaluation metric strictly aligned with minimization of Total Classification Cost (TCC). Grounded in sample-level cost modeling, we develop a scalable, cost-sensitive evaluation framework applicable to any metric expressible as a linear combination of sample-dependent quantities. Contribution/Results: We formally characterize the limited validity of rebalancing techniques—showing they are only optimal under specific cost structures—and introduce a WA estimation method that requires no complete prior knowledge of misclassification costs. Extensive experiments across diverse datasets, distribution shifts, and deployment scenarios demonstrate WA’s high correlation with and robustness to TCC, significantly enhancing evaluation comparability and practical utility.
📝 Abstract
Selecting an appropriate evaluation metric for classifiers is crucial for model comparison and parameter optimization, yet there is not consensus on a universally accepted metric that serves as a definitive standard. Moreover, there is often a misconception about the perceived need to mitigate imbalance in datasets used to train classification models. Since the final goal in classifier optimization is typically maximizing the return of investment or, equivalently, minimizing the Total Classification Cost (TCC), we define Weighted Accuracy (WA), an evaluation metric for binary classifiers with a straightforward interpretation as a weighted version of the well-known accuracy metric, coherent with the need of minimizing TCC. We clarify the conceptual framework for handling class imbalance in cost-sensitive scenarios, providing an alternative to rebalancing techniques. This framework can be applied to any metric that, like WA, can be expressed as a linear combination of example-dependent quantities and allows for comparing the results obtained in different datasets and for addressing discrepancies between the development dataset, used to train and validate the model, and the target dataset, where the model will be deployed. It also specifies in which scenarios using UCCs-unaware class rebalancing techniques or rebalancing metrics aligns with TCC minimization and when it is instead counterproductive. Finally, we propose a procedure to estimate the WA weight parameter in the absence of fully specified UCCs and demonstrate the robustness of WA by analyzing its correlation with TCC in example-dependent scenarios.