🤖 AI Summary
This study addresses the inconsistency of existing prediction evaluation metrics—such as ABC and Gini—with the principle of mean consistency, stemming from their reliance on predicted values for weighting, which can lead to erroneous model selection. Building upon Bregman divergences, the authors develop a mean-consistent loss framework, rederive the Murphy decomposition to disentangle prediction error into calibration and discrimination components, and establish a theoretical link between these components and Lorenz-curve-based metrics. They propose a new metric, ABC², to enhance sensitivity to mean calibration, and demonstrate that ABC, ABC², and Gini all violate mean consistency due to prediction-dependent weighting. Furthermore, they prove the equivalence between the number of crossings in Lorenz and Murphy curves and, under a single-crossing condition, provide a weak dominance criterion for predictive superiority, offering both theoretical grounding and practical guidelines for reliable model evaluation.
📝 Abstract
We study the evaluation of real-valued point predictors under the decision-theoretic framework of mean-consistent loss functions given by the Bregman divergences. We first derive a new version of Murphy's decomposition of the expected loss which does not directly include the response itself but only its predictors. We then relate the miscalibration and the discrimination component of the Murphy's decomposition to Lorenz-curve-based accuracy measures that are widely used in practice. Besides the usual area between the concentration and Lorenz curves, ABC, we introduce a mean-squared version ABC$^2$ that mitigates some of the weaknesses of the original ABC in identifying mean-calibration. More importantly, both ABC and ABC$^2$ are shown to rely on predictor-dependent weights, so they fail to align with the class of mean-consistent scoring functions. In the same spirit, we derive a similar result for the widely used Gini score. These results indicate that ABC, ABC$^2$ and Gini scores may lead to dishonest evaluation of point predictions when used for model selection; this gives support to use mean-consistent loss functions as well as the miscalibration and the discrimination measure from the Murphy's decomposition of the expected loss for model evaluation. Finally, we study forecast dominance when Lorenz curves intersect. We show that Lorenz and Murphy's curves have the same number of crossings and, in the one-crossing case, we establish weaker dominance criteria for subclasses of Bregman divergences through third-degree stochastic dominance.