🤖 AI Summary
This study addresses the rare-event validation bias in electronic health record (EHR) algorithm evaluation caused by prohibitive manual review costs. We propose a sampling-based statistical validation framework that exhaustively verifies positive samples while randomly sampling negative ones. The method incorporates Horvitz-Thompson estimators with finite population variance corrections to estimate performance metrics, and integrates Neyman optimal allocation with adaptive sampling techniques to support confidence interval planning and risk-guided sample allocation. Experimental results demonstrate that this framework substantially reduces validation bias, achieves near-nominal coverage rates, and proves effective in a breast cancer recurrence prediction task.
📝 Abstract
Electronic health record (EHR) phenotyping algorithms support clinical prediction, observational research, and quality improvement, but manual chart review for gold-standard validation is costly. For algorithms targeting rare outcomes, algorithm-negative patients often greatly outnumber algorithm-positive patients. We propose a design-based validation framework that fully verifies algorithm-positive patients and reviews a simple random sample (SRS) of algorithm-negative patients. Horvitz-Thompson estimators of false-negative (FN) and true-negative totals yield estimates of sensitivity, specificity, negative predictive value, positive predictive value, and accuracy. We derive finite-population-corrected variance estimators and confidence intervals (CIs), with sample-size planning for a prespecified CI half-width. An optional Phase 2 extension uses pilot data to assess whether risk-guided Neyman allocation may improve precision. In simulations, the design-based approach substantially reduced bias relative to naive verified-sample analyses and achieved near-nominal CI coverage. Adaptive allocation reduced root mean squared error and CI width when FNs were concentrated across risk strata, but offered little benefit when the risk signal was weak. We illustrate the framework using a breast cancer recurrence algorithm in 3,336 patients, with endpoint-specific algorithm-negative review budgets of 125, 150, and 175 charts. The framework combines SRS-based validation and precision planning with optional pilot-guided adaptive allocation.