Multi-Site Real-World Performance of Commercial AI for Pulmonary and Incidental Pulmonary Embolism Detection

2026-09-29

AI Summary

The real-world, multicenter performance of artificial intelligence models for pulmonary embolism detection remains unclear. This study addresses this gap by leveraging large language models to automatically extract labels from radiology reports as a reference standard, enabling a large-scale retrospective validation of two FDA-cleared AI algorithms across 17 institutions using CT pulmonary angiography and routine contrast-enhanced CT scans. Results indicate that while model specificity exceeded baseline expectations, sensitivity was suboptimal, with notably high miss rates for subsegmental and non-acute emboli. By innovatively integrating LLM-based automated annotation into post-market evaluation, this work highlights the critical necessity for continuous surveillance of AI systems in clinical practice to ensure sustained diagnostic reliability across diverse real-world settings.

Abstract

Pulmonary embolism (PE) is a leading cause of cardiovascular mortality, yet the real-world performance of FDA-cleared AI detection models remains incompletely characterized. We retrospectively evaluated two FDA-cleared AI algorithms from a single commercial platform (Aidoc Medical BriefCase), one for PE triage on dedicated CT pulmonary angiography (CTPA; n = 30,678) and one for incidental PE (iPE) detection on routine contrast-enhanced CTs (n = 37,191), across a 17-facility academic health system. Reference-standard labels were extracted from radiology reports using a validated LLM pipeline (97% accuracy, kappa = 0.94). The PE model achieved 86.8% sensitivity and 99.1% specificity, with sensitivity declining from 99.3% for saddle emboli to 72.9% for subsegmental PE, and from 89.7% for acute to 65.3% for non-acute PE. The iPE model achieved 73.5% sensitivity and 99.8% specificity. Both models demonstrated lower sensitivity than FDA-clearance benchmarks while exceeding cleared specificity, with diminishing performance for peripheral and non-acute emboli mirroring known human reader limitations and underscoring the need for standardized post-market surveillance of AI-enabled medical devices.

Problem

• Pulmonary Embolism
• Artificial Intelligence
• Real-World Performance
• Incidental Pulmonary Embolism
• Post-market Surveillance

Innovation

• Real-world performance
• Large language model
• Pulmonary embolism detection
• Incidental pulmonary embolism
• Post-market surveillance