MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models

2026-06-04

AI Summary

This work addresses the lack of systematic evaluation of fine-grained visual perception capabilities in current vision-language models for biomedical image understanding, as existing benchmarks cover only limited modalities and tasks. To bridge this gap, the authors introduce MMBU, a large-scale multimodal biomedical understanding benchmark encompassing 35 imaging submodalities and structured metadata, supporting tasks such as non-localized classification, localized classification, and object detection. Notably, MMBU features both open-ended and closed-ended variants to comprehensively assess model perception and generalization across diverse biological scales, clinical contexts, and imaging modalities. Evaluation of 15 state-of-the-art models reveals that, despite modest gains from medical adaptation, current models still exhibit significant deficiencies in fine-grained perception and cross-domain generalization within real-world biomedical settings.

Abstract

Vision and language models (VLMs) hold immense promise to transform biomedical imaging workflows, from detecting lesions in chest X-rays to profiling cellular features in microscopy. Realizing this potential, however, requires robust and fine-grained visual perception. Models need to correctly interpret subtle features in images, and they must do so across diverse biomedical modalities, scales, and contexts. Nevertheless, current benchmarks remain limited. To address these gaps, we introduce the Massive Multimodal Biomedical Understanding (MMBU) benchmark. It is the largest biomedical vision and language benchmark to date, covering 35 submodalities with rich structured metadata. It includes both open and closed versions of ungrounded classification, grounded classification, and object detection, enabling systematic evaluation of model performance across biological scales, clinical settings, and imaging modalities. Evaluating 15 open-weight and 2 frontier VLMs, we find that while medical adaptation provides measurable gains for some models, the high accuracy often reported on established benchmarks can mask deficiencies in visual perception and domain generalization.

Problem

• vision-language models
• biomedical imaging
• visual perception
• benchmark
• domain generalization

Innovation

• vision-language models
• biomedical benchmark
• multimodal understanding
• visual perception
• domain generalization