Confident but Unreliable: A Behavioral Safety Audit of Vision-Language Models on Brain MRI

2026-08-03

AI Summary

This study addresses the critical issue of overconfident yet unreliable predictions in vision-language models (VLMs) applied to medical imaging tasks. The authors present the first systematic audit of six instruction-tuned VLMs on a dataset of 4,102 brain MRI scans, leveraging publicly available metadata and expert-derived segmentation masks for automated annotation. Using a multidimensional evaluation framework—including expected calibration error (ECE), confidence analysis, and open-ended diagnostic assessments—they reveal severe miscalibration across all models: erroneous responses exhibit average confidences of 0.82–0.97, with 33%–46% classified as high-confidence errors. Notably, domain-specific fine-tuning on medical data fails to mitigate this unreliability. The findings underscore significant limitations in current evaluation protocols and advocate for the inclusion of confidence reliability and high-confidence error rate as standard metrics in the assessment of medical VLMs.

Abstract

Vision-language models (VLMs), including medical specialists, are increasingly proposed for medical imaging, yet their stated confidence is rarely evaluated separately from correctness. We use brain MRI as a controlled, high-stakes testbed for a broader failure mode in frontier multimodal systems: models can appear competent while lacking reliable self-knowledge. We present an automatically graded behavioral audit and pilot study of six instruction-tuned VLMs (five general-purpose and one medical specialist) on 4,102 images (4,032 axial/coronal/sagittal MRI slices from 250 subjects plus 70 non-brain/noise controls), with labels derived from public metadata and released expert segmentation masks rather than new human annotation. Across models, answer coverage is near-complete, but verbalized-confidence calibration is poor: ECE ranges from 0.27 to 0.40, mean confidence on incorrect answers ranges from 0.82 to 0.97, and 33-46% of answered items are high-confidence errors. The most accurate model is also the most confident on its errors, while a base/specialist family contrast suggests that medical adaptation improves tumor-presence detection without improving confidence reliability. Open-ended diagnostics further show that hallucination and abstention vary separately from multiple-choice accuracy. These findings argue that medical-image VLM evaluation should report verbalized-confidence reliability, confident error, hallucination, and abstention alongside accuracy.

Problem

• vision-language models
• confidence calibration
• medical imaging
• brain MRI
• reliability

Innovation

• confidence calibration
• behavioral audit
• vision-language models
• medical imaging
• hallucination