🤖 AI Summary
Current clinically deployed AI systems for pulmonary nodule detection lack post-deployment validation against variations in local CT acquisition parameters, potentially leading to substantial performance deviations from benchmark expectations. This work proposes a reproducible validation framework suitable for resource-constrained settings that does not require proprietary scanner data. The approach leverages physics-based image-domain simulations: Gaussian noise models reduced radiation dose, while z-axis moving averages emulate slice thickness alterations. Sensitivity is evaluated using a 15-mm matching criterion. Experiments reveal that slice thickness exerts a far greater impact on model performance than noise—increasing thickness to 5 mm reduces the sensitivity of MONAI RetinaNet from a baseline of 45.2% to 26.2% (a 42% relative drop), whereas a 50% dose reduction only mildly decreases sensitivity to 42.1%. Notably, some cases already fail at baseline, underscoring the critical need for post-deployment validation.
📝 Abstract
Background: Artificial intelligence (AI) assisted lung nodule detection systems are increasingly deployed in clinical settings without site-specific validation. Performance reported under benchmark conditions may not reflect real-world behavior when acquisition parameters differ from training data. Purpose: To propose and demonstrate a physics-guided framework for evaluating the sensitivity of a deployed lung nodule detection model to systematic variation in CT acquisition parameters. Methods: Twenty-one cases from the publicly available LIDC-IDRI dataset were evaluated using a MONAI RetinaNet model pretrained on LUNA16 (fold 0, no fine-tuning). Five imaging conditions were tested: baseline, 25% dose reduction, 50% dose reduction, 3 mm slice thickness, and 5 mm slice thickness. Dose reduction was simulated via image-domain Gaussian noise; slice thickness via moving average along the z-axis. Detection sensitivity was computed at a confidence threshold of 0.5 with a 15 mm matching criterion. Results: Baseline sensitivity was 45.2% (57/126 consensus nodules). Dose reduction produced slight degradation: 41.3% at 25% dose and 42.1% at 50% dose. The 5 mm slice thickness condition produced a marked drop to 26.2% - a 19 percentage point reduction representing a 42% relative decrease from baseline. This finding was consistent across confidence thresholds from 0.1 to 0.9. Per-case analysis revealed heterogeneous performance including two cases with complete detection failure at baseline. Conclusion: Slice thickness represents a more fundamental constraint on AI detection performance than image noise under the conditions tested. The proposed framework is reproducible, requires no proprietary scanner data, and is designed to serve as the basis for ongoing post-deployment QA in resource-constrained environment.