assess clinical relevance

Existing automatic evaluation metrics achieve high scores on radiology report generation tasks yet lack clinical trust due to fundamental deficiencies in clinical semantic understanding—specifically, their inability to distinguish clinically significant errors, over-penalization of harmless lexical or syntactic variations, and non-monotonic responses to error severity. Method: We propose the first clinically oriented meta-evaluation framework, defining dual dimensions—clinical alignment and core capability—and constructing a manually annotated dataset featuring fine-grained error types, clinical importance ratings, and expert explanations. Contribution/Results: Through systematic benchmarking of mainstream metrics, we empirically expose their semantic blind spots. Our framework not only precisely diagnoses failure mechanisms of current metrics but also establishes a methodological foundation and practical paradigm for trustworthy, interpretable evaluation of medical text generation.

Current health AI evaluation benchmarks lack standardized descriptions of user queries, limiting their ability to accurately reflect model applicability in real-world clinical settings. This study systematically identifies this “validity gap” and proposes adapting clinical trial reporting standards to create structured query profiles. Leveraging large language models, we automatically annotated 18,707 health-related queries from six public benchmarks using a 16-dimensional taxonomy capturing clinical context, topic, and intent. Our analysis reveals significant structural biases: existing benchmarks severely underrepresent complex diagnostic information such as laboratory tests, imaging, and raw clinical notes; safety-critical scenarios (e.g., self-harm) constitute less than 0.7% of queries; and coverage of pediatric, geriatric, and chronic disease populations is markedly insufficient—highlighting a substantial misalignment between current evaluation frameworks and actual clinical needs.

This work addresses the mismatch between conventional machine learning practices and the specific performance requirements of clinical tasks in healthcare settings. Traditional approaches rely on differentiable validation losses for model optimization, which often fail to align with clinically meaningful outcomes. To bridge this gap, the paper proposes replacing standard loss functions with non-differentiable yet clinically interpretable custom metrics to guide critical optimization decisions—such as hyperparameter selection and training termination—thereby redefining the model validation pipeline. In two controlled experiments, models optimized using this framework demonstrated significantly superior performance on key clinical tasks compared to those guided by conventional differentiable validation losses. This approach overcomes the inherent limitation of relying solely on differentiable objectives and better aligns medical AI development with real-world clinical goals.

In chest X-ray machine learning, commonly used automatic evaluation metrics—such as report-derived labels or generic image quality assessment (IQA) measures—often fail to accurately reflect clinical judgment, leading to biased performance evaluations. This study systematically investigates how different reference standards affect model performance and ranking in both pathology classification and image quality assessment. Leveraging expert annotations alongside public datasets, we compare multiple supervised classifiers (e.g., ResNet, DenseNet) and vision–language models (e.g., MedKLIP, GLoRIA, ConVIRT). We demonstrate for the first time that the choice of evaluation reference substantially alters model rankings and performance interpretations. Furthermore, widely adopted IQA metrics like SSIM and PSNR frequently diverge from expert assessments of diagnostic usability, underscoring the necessity of aligning evaluation criteria with clinical validity as a core component of model validation.

Current medical AI evaluation benchmarks predominantly emphasize knowledge acquisition, failing to adequately capture model reliability, safety, and clinical utility in real-world settings. To address this gap, this work proposes the first systematic evaluation framework aligned with clinical workflows, encompassing end-to-end tasks such as clinical documentation, decision support, and administrative processes. The framework integrates authentic multimodal clinical data and introduces task-specific metrics to comprehensively assess generative models, multimodal systems, and AI agents. Empirical results reveal a substantial performance gap between state-of-the-art models on real-world tasks and their scores on medical knowledge exams—scoring 0.74–0.85 in documentation, 0.61–0.76 in clinical decision-making, and 0.53–0.63 in administrative tasks—highlighting the limitations of existing evaluation paradigms and underscoring the critical role of this framework in advancing the clinical deployment of medical AI.

This study addresses the challenge of ensuring rigor in causal inference under multi-source heterogeneous data fusion by proposing a structured design paradigm grounded in the target trial framework. The approach explicitly incorporates the target population and its sampling model into the causal analysis, systematically integrating external controls, generalizability, and transportability assessments through data element alignment, transparent assumption articulation, and emulation of the target trial. Its key innovation lies in anchoring the entire framework to a precise definition of the target population, thereby identifying and mitigating irreconcilable conflicts across data sources. This strategy enhances both the reliability and interpretability of causal conclusions derived from complex, real-world data ecosystems.

This study addresses the widespread yet unverified assumption of component consistency in medical imaging AI benchmarks, which often undermines result reproducibility. Focusing on an archived chest X-ray vision-language model benchmark, the authors conduct a systematic reproducibility audit without re-invoking models or adding new annotations. Through DICOM metadata analysis, prompt binding tracing, automated label extraction, statistical code replication, and Holm-corrected multiple hypothesis testing, they uncover critical technical and metadata issues—including missing image polarity inversion, erroneous dataset splits, and truncated radiology reports. Reconstructing the evaluation cohort substantially alters the original statistical conclusions, leading to the retraction of prior performance claims and clinical assertions. The work concludes by proposing machine-verifiable control protocols to enhance the reliability of future benchmarks.

This study addresses the critical issue of hallucinations in medical imaging AI—such as fabricated anatomical structures, left-right confusion, or erroneous measurements—which can lead to misdiagnosis and inappropriate treatment. The work proposes the first cross-modal hallucination taxonomy encompassing the entire imaging pipeline, systematically evaluating hallucination tendencies in both general-purpose and medical-specific foundation models. It integrates physical constraints, chain-of-thought prompting, and human-in-the-loop mechanisms to develop a detection and mitigation strategy aligned with FDA’s full lifecycle regulatory requirements. Key findings reveal that medical-specific models, due to overfitting, are paradoxically more prone to hallucinations; that combining multiple mitigation strategies effectively addresses diverse failure modes; and that radiologist review remains essential for safe clinical deployment.