🤖 AI Summary
The impact of class imbalance correction on model discriminative performance and probability calibration in clinical risk prediction remains unclear. This study systematically evaluates the effects of SMOTE, random oversampling (ROS), and random undersampling (RUS) across ten real-world clinical datasets using a range of linear and nonlinear models. Comprehensive comparisons are conducted using metrics including ROC-AUC, Brier score, and calibration intercept/slope. Results indicate that none of the three resampling methods significantly improve discrimination, yet all consistently degrade probability calibration—evidenced by increased Brier scores (0.029–0.080) and substantial shifts in calibration parameters—revealing systematic distortion in predicted risk estimates. These findings challenge the conventional use of resampling techniques in clinical prediction modeling.
📝 Abstract
Objective: ML-based clinical risk prediction models are increasingly used to support decision-making in healthcare. While class-imbalance correction techniques are commonly applied to improve model performance in settings with rare outcomes, their impact on probabilistic calibration remains insufficiently understood. This study evaluated the effect of widely used resampling strategies on both discrimination and calibration across real-world clinical prediction tasks. Methods: Ten clinical datasets spanning diverse medical domains and including 605,842 patients were analyzed. Multiple machine-learning model families, including linear models and several non-linear approaches, were evaluated. Models were trained on the original data and under three commonly used 1:1 class-imbalance correction strategies (SMOTE, RUS, ROS). Performance was assessed on held-out data using discrimination and calibration metrics. Results: Across all datasets and model families, resampling had no positive impact on predictive performance. Changes in the Receiver Operating Characteristic Area Under Curve (ROC-AUC) relative to models trained on the original data were small and inconsistent (ROS: -0.002, p<0.05; RUS: -0.004, p>0.05; SMOTE: -0.01, p<0.05), with no resampling strategy demonstrating a systematic improvement. In contrast, resampling in general degraded the calibration performance. Models trained using imbalance correction exhibited higher Brier scores (0.029 to 0.080, p<0.05), reflecting poorer probabilistic accuracy, and marked deviations in calibration intercept and slope, indicating systematic distortions of predicted risk despite preserved rank-based performance. Conclusion: In a diverse set of real-world clinical prediction tasks, commonly used class-imbalance correction techniques did not provide generalizable improvements in discrimination and were associated with degraded calibration.