Medical-Checklist: Assessing the Comprehension of Medical Images by Multimodal Models

2026-07-24

AI Summary

Although existing medical multimodal models demonstrate strong performance on vision–language tasks, their capacity to accurately comprehend the semantic alignment between medical images and associated text remains inadequately evaluated. To address this gap, this work introduces Medical-Checklist, a benchmark that constructs pairs of correct and incorrect captions differing only in a single medical concept, requiring models to perform binary discrimination. This design enables precise assessment of fine-grained medical image understanding while mitigating dataset bias. The benchmark further supports unified evaluation across medical subdomains and out-of-distribution generalization. Experimental results reveal that current state-of-the-art models exhibit significant deficiencies in fundamental medical image comprehension, highlighting critical challenges that must be resolved before clinical deployment.

Abstract

This paper introduces a new benchmark test, Medical-Checklist, for assessing medical multimodal models. The recent advancements in multimodal models have demonstrated significant potential in the field of medical vision-language tasks. However, it is becoming increasingly clear that evaluating these models' performance, whether they are applied to natural or medical images, is challenging. The critical question is whether the models can accurately understand an input image while associating it with relevant input text. To address this, Medical-Checklist imposes a binary test on the models: they are given an image and two captions, where one is correct and the other incorrect, and the model must select the correct one. The incorrect caption contains a single medical concept (word or phrase) that is inaccurately substituted from the correct caption. Although the task is simple, this simplicity enables the unified assessment of diverse multimodal models designed and learned on different principles. It also enables us to verify whether models correctly understand a wide range of medical concepts across various medical sub-domains. Medical-Checklist is designed to reduce potential biases in data and to enable evaluation of the models' ability to handle out-of-distribution inputs, which were difficult in existing datasets. When evaluating four state-of-the-art medical multimodal models with Medical-Checklist, it was revealed that despite their excellent performance in specific tasks such as Med-VQA, they may not correctly understand images, suggesting a long journey ahead for clinical application. The dataset and code will be made public upon acceptance.

Problem

• medical multimodal models
• image-text comprehension
• model evaluation
• medical concept understanding
• benchmarking

Innovation

• Medical-Checklist
• multimodal models
• medical image understanding
• binary caption selection
• out-of-distribution evaluation