MedErr-CT: A Visual Question Answering Benchmark for Identifying and Correcting Errors in CT Reports

2025-06-23

AI Summary

Current medical visual question answering (VQA) benchmarks lack clinical relevance and fail to assess the higher-order reasoning capabilities of multimodal large language models (MLLMs) in detecting and correcting errors in CT reports. To address this, we introduce MedErr-CT—the first fine-grained VQA benchmark specifically designed for CT report error correction—covering six clinically authentic error types and establishing a three-tiered task hierarchy: classification, detection, and correction. We propose a novel cross-modal (vision + text) error annotation framework, integrating 3D medical MLLMs, cross-modal alignment, and multi-level reasoning for joint image–report analysis. Comprehensive experiments on state-of-the-art 3D medical MLLMs reveal significant performance disparities across error categories, thereby filling a critical gap in clinical error-correction evaluation. MedErr-CT provides both a quantifiable assessment standard and high-quality data to enhance diagnostic accuracy and model reliability.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Computed Tomography (CT) plays a crucial role in clinical diagnosis, but the growing demand for CT examinations has raised concerns about diagnostic errors. While Multimodal Large Language Models (MLLMs) demonstrate promising comprehension of medical knowledge, their tendency to produce inaccurate information highlights the need for rigorous validation. However, existing medical visual question answering (VQA) benchmarks primarily focus on simple visual recognition tasks, lacking clinical relevance and failing to assess expert-level knowledge. We introduce MedErr-CT, a novel benchmark for evaluating medical MLLMs' ability to identify and correct errors in CT reports through a VQA framework. The benchmark includes six error categories - four vision-centric errors (Omission, Insertion, Direction, Size) and two lexical error types (Unit, Typo) - and is organized into three task levels: classification, detection, and correction. Using this benchmark, we quantitatively assess the performance of state-of-the-art 3D medical MLLMs, revealing substantial variation in their capabilities across different error types. Our benchmark contributes to the development of more reliable and clinically applicable MLLMs, ultimately helping reduce diagnostic errors and improve accuracy in clinical practice. The code and datasets are available at https://github.com/babbu3682/MedErr-CT.

Problem

• Identifying and correcting errors in CT reports
• Assessing medical MLLMs' error detection capabilities
• Evaluating clinical relevance of medical VQA benchmarks

Innovation

• Introduces MedErr-CT benchmark for CT report errors
• Includes six error categories and three tasks
• Evaluates 3D medical MLLMs for diagnostic accuracy