Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks

2024-10-24

arXiv.org

AI Summary

Current medical multimodal large language models lack explicit anatomical region awareness in medical images, hindering their ability to emulate clinicians’ diagnostic reasoning—namely, “global observation followed by local focus.” To address this, we propose MedRegA, the first region-aware, bilingual general-purpose medical multimodal LLM, supporting eight imaging modalities and both image-level and anatomical region-level vision-language tasks. Our method introduces a novel region-centered task paradigm, constructs the large-scale bilingual and interpretable MedRegInstruct dataset, and integrates region-aware visual encoding, anatomical structure detection alignment, multi-source joint training, and bilingual instruction tuning. Evaluated on visual question answering, radiology report generation, and image classification, MedRegA achieves state-of-the-art performance. Notably, it is the first to enable region-localization-guided generation and structured reasoning, significantly enhancing model interpretability and clinical interactivity.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Several medical Multimodal Large Languange Models (MLLMs) have been developed to address tasks involving visual images with textual instructions across various medical modalities, achieving impressive results. Most current medical generalist models are region-agnostic, treating the entire image as a holistic representation. However, they struggle to identify which specific regions they are focusing on when generating a sentence. To mimic the behavior of doctors, who typically begin by reviewing the entire image before concentrating on specific regions for a thorough evaluation, we aim to enhance the capability of medical MLLMs in understanding anatomical regions within entire medical scans. To achieve it, we first formulate Region-Centric tasks and construct a large-scale dataset, MedRegInstruct, to incorporate regional information into training. Combining our collected dataset with other medical multimodal corpora for training, we propose a Region-Aware medical MLLM, MedRegA, which is the first bilingual generalist medical AI system to simultaneously handle image-level and region-level medical vision-language tasks across a broad range of modalities. Our MedRegA not only enables three region-centric tasks, but also achieves the best performance for visual question answering, report generation and medical image classification over 8 modalities, showcasing significant versatility. Experiments demonstrate that our model can not only accomplish powerful performance across various medical vision-language tasks in bilingual settings, but also recognize and detect structures in multimodal medical scans, boosting the interpretability and user interactivity of medical MLLMs. Our project page is https://medrega.github.io.

Problem

• Enhance medical MLLMs' anatomical region understanding in scans
• Develop bilingual AI for image and region-level medical tasks
• Improve interpretability and interactivity in medical vision-language models

Innovation

• Region-Aware bilingual medical MLLM
• Large-scale dataset MedRegInstruct
• Handles image and region-level tasks