🤖 AI Summary
This study addresses the limitation of the traditional Brier score, which conflates calibration and discrimination in probabilistic forecasting, thereby hindering targeted optimization. The authors propose the Manokhin probability matrix, which for the first time decouples predictive quality into two orthogonal dimensions—calibration and discrimination—by constructing a two-dimensional diagnostic framework based on the Spiegelhalter Z-statistic and the expected rank interpretation of AUC-ROC. This framework categorizes classifiers into four archetypes: Eagle, Bull, Sloth, and Mole, and reveals a theoretical asymmetry: discrimination is inherently difficult to improve, whereas calibration can be effectively post-processed. Consequently, the paper advocates a practical guideline of “optimize discrimination first, then calibrate.” Large-scale evaluation on the TabArena-v0.1 benchmark across 21 classifiers and 5 calibrators shows that Venn-Abers calibration reduces log-loss by 6.5–12.6% for Bull-type models but slightly degrades Eagle-type performance, confirming an inherent trade-off between calibration and discrimination.
📝 Abstract
The Brier score conflates two distinct properties of probabilistic predictions: reliability (calibration error) and resolution (discriminatory power). We introduce the Manokhin Probability Matrix, a BCG-style two-dimensional diagnostic framework that separates them. Classifiers are placed on a 2x2 grid by Spiegelhalter Z-statistic and AUC-ROC expected rank, then assigned to one of four archetypes: Eagle (good on both axes), Bull (strong discrimination, poor calibration), Sloth (well-calibrated, weak discriminator), and Mole (poor on both). Each archetype carries a distinct prescription. We populate the matrix from a large-scale empirical study spanning 21 classifiers, 5 post-hoc calibrators, and 30 real-world binary classification tasks from the TabArena-v0.1 suite. The assignment is unambiguous. CatBoost, TabICL, EBM, TabPFN, GBC, and Random Forest are Eagles. XGBoost, LightGBM, and HGB are Bulls; Venn-Abers calibration cuts log-loss by 6.5 to 12.6% on Bulls but degrades Eagles by 2.1%. SVM, LR, LDA, and the empirical base-rate predictor are Sloths. MLP, KNN, Naive Bayes, and ExtraTrees are Moles. A theoretical asymmetry follows: no order-preserving post-hoc calibrator can add discriminatory power (Proposition 1), so calibration is the fixable part and discrimination is the hard part. The practical rule is direct: do not optimise aggregate Brier score without first decomposing it; optimise discrimination first, then fix calibration post-hoc. Code and raw experimental data are available at https://github.com/valeman/classifier_calibration.