🤖 AI Summary
Current sample size calculations for clinical prediction models ensure only that performance metrics meet target values *on average*, neglecting sampling variability—resulting in unstable model performance and low probability of achieving acceptable performance (PrAP) in practice. This paper proposes a novel sample size determination framework centered on PrAP, formally adopting “probability of attaining acceptable performance” as the primary statistical objective—replacing conventional expectation-based approaches. Through simulation studies and analytical derivations, we develop robust methods for estimating calibration slope in binary outcome settings, implemented in the R package `samplesizedev`. Results demonstrate that conventional methods yield PrAPs typically below 60%, whereas our approach consistently achieves PrAPs exceeding 80%, with particularly pronounced gains when fewer predictors are included. This substantially improves model reliability and reproducibility.
📝 Abstract
Existing approaches to sample size calculations for developing clinical prediction models have focused on ensuring that the expected value of a chosen performance measure meets a pre-specified target. For example, to limit model-overfitting, the sample size is commonly chosen such that the expected calibration slope (CS) is 0.9, close to 1 for a perfectly calibrated model. In practice, due to sampling variability, model performance can vary considerably across different development samples of the recommended size. If this variability is high, the probability of obtaining a model with performance close to the target for a given measure may be unacceptably low. To address this, we propose an adapted approach to sample size calculations that explicitly incorporates performance variability by targeting the probability of acceptable performance (PrAP). For example, in the context of calibration, we may define a model as acceptably calibrated if CS falls in a pre-defined range, e.g. between 0.85 and 1.15. Then we choose the required sample size to ensure that PrAP(CS)=80%. For binary outcomes we implemented our approach for CS within a simulation-based framework via the R package `samplesizedev'. Additionally, for CS specifically, we have proposed an equivalent analytical calculation which is computationally efficient. While we focused on CS, the simulation-based framework is flexible and can be easily extended to accommodate other performance measures and types of outcomes. When adhering to existing recommendations, we found that performance variability increased substantially as the number of predictors, p, decreased. Consequently, PrAP(CS) was often low. For example, with 5 predictors, PrAP(CS) was around 50%. Our adapted approach resulted in considerably larger sample sizes, especially for p<10. Applying shrinkage tends to improve PrAP(CS).