Rethinking Clinical Relevance in Chest X-ray Machine Learning: How Evaluation References Define Performance

2026-07-28

AI Summary

In chest X-ray machine learning, commonly used automatic evaluation metrics—such as report-derived labels or generic image quality assessment (IQA) measures—often fail to accurately reflect clinical judgment, leading to biased performance evaluations. This study systematically investigates how different reference standards affect model performance and ranking in both pathology classification and image quality assessment. Leveraging expert annotations alongside public datasets, we compare multiple supervised classifiers (e.g., ResNet, DenseNet) and vision–language models (e.g., MedKLIP, GLoRIA, ConVIRT). We demonstrate for the first time that the choice of evaluation reference substantially alters model rankings and performance interpretations. Furthermore, widely adopted IQA metrics like SSIM and PSNR frequently diverge from expert assessments of diagnostic usability, underscoring the necessity of aligning evaluation criteria with clinical validity as a core component of model validation.

Abstract

Chest X-ray (CXR) machine learning relies heavily on automated evaluation using reference standards that aim to approximate clinical judgment. However, commonly used report-derived labels for pathology classification or generic image quality metrics for reconstruction may not reliably reflect clinical judgment. We systematically investigate how evaluation-reference choices affect model performance and ranking in both pathology classification and image quality assessment (IQA). To enable controlled comparison across evaluation references, we collected paired expert image- and report-derived labels for thoracic findings from a clinical cohort at Cambridge University Hospitals (CUH) and curated a subset of the public MIMIC-CXR dataset, along with expert ratings of diagnostic image quality. We show that for supervised image classifiers (ResNet, DenseNet), several zero-shot and fine-tuned vision-language models (e.g., MedKLIP, GLoRIA, and ConVIRT), changing the label source leads to substantial differences not only in performance estimates but also in model rankings. In parallel, alignment of IQA measures with expert judgment depends heavily on the choice of measure, and commonly used IQA metrics such as SSIM and PSNR often fail to align with expert assessments of diagnostic usability. Our results demonstrate that evaluation choices are crucial: they can determine which models and methods appear best and are therefore selected for further development or deployment. The selection of evaluation references should therefore be treated as a central component of clinical validity in CXR machine learning, and justified with respect to the pathology, imaging task, and intended downstream clinical use.

Problem

• chest X-ray
• clinical relevance
• evaluation reference
• image quality assessment
• pathology classification

Innovation

• evaluation reference
• clinical relevance
• chest X-ray
• image quality assessment
• model ranking