MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models

2026-06-04

AI Summary

Existing large vision-language models in medical imaging struggle to simultaneously achieve visual-language understanding and pixel-level image segmentation, hindering the integration of visual findings with semantic interpretation in clinical reasoning. To address this limitation, this work proposes MedSIGHT, a unified framework that employs a region-aware encoder to generate region-centric tokens and leverages a medical region codebook as a discrete vocabulary for the language model, enabling end-to-end spatial grounding and segmentation reconstruction. By innovatively integrating region encoding with a large language model and combining progressive training with multimodal instruction fine-tuning, MedSIGHT attains state-of-the-art performance across diverse medical imaging modalities in both visual-language understanding and segmentation tasks—using only 72,000 instruction samples.

Abstract

Medical large vision-language models (Med-LVLMs) have recently achieved remarkable progress in vision-language comprehension and medical image segmentation. However, existing models still struggle to unify these two capabilities, which is essential for achieving clinically reasoning that connects visual findings with semantic interpretation. We present MedSIGHT, a unified framework that equips Med-LVLMs with structured, pixel-level understanding for grounded visual comprehension. MedSIGHT introduces a novel Region Perceiver module that produces region-centric tokens, encoding spatial information directly into representation space of the language model. We further propose a medical region codebook into the LLM vocabulary, allowing the model to generate discrete region codes as symbolic representations of anatomical and pathological regions. These codes are decoded through the Region Perceiver to reconstruct segmentation mask, achieving end-to-end spatial grounding. Lastly, MedSIGHT combines Region Perceiver, Codebook and LLM using our proposed progressive training strategy to gradually aligns these modules stably. Trained on only 72K multimodal instruction pairs, MedSIGHT achieves state-of-the-art performance across diverse imaging modalities on both medical comprehension and segmentation tasks.

Problem

• medical large vision-language models
• visual comprehension
• image segmentation
• spatial grounding
• clinical reasoning

Innovation

• Region Perceiver
• medical region codebook
• grounded visual comprehension
• unified vision-language framework
• progressive training strategy