Localization Lens for Improving Medical Vision-Language Models

2026-10-03

International Conference on Medical Image Computing and Computer-Assisted Intervention

AI Summary

This study addresses the limitations of medical vision-language models (Med-VLMs) in anatomical comprehension and spatial localization by proposing a model-agnostic, localization-aware enhancement framework. The method introduces a "localization lens" across the data, architecture, and alignment levels, integrating pixel shuffle techniques with a decoupled contrastive loss (DCL) to systematically improve spatial reasoning capabilities. Furthermore, an expert-validated representation mechanism is incorporated to enhance the robustness of anatomical and spatial understanding. Experimental results demonstrate that the proposed framework can be seamlessly integrated into diverse Med-VLM architectures, yielding substantial performance gains with up to a 6.2% improvement in accuracy on medical visual question answering tasks.

Abstract

Medical Vision-Language Models (Med-VLMs) have demonstrated strong capabilities in clinical tasks. However, they often struggle to understand anatomical structures and spatial positioning, which are crucial for medical reasoning. To address this, we propose a localization-aware enhancement to the Med-VLM pipeline, introducing improvements at three levels: data,architecture, and alignment. First, we introduce localization lens, a set of expert-validated representations that provide richer anatomical and positional context. However, as these representations increase input complexity, we integrate pixel shuffle within the model architecture to filter and refine representations, enhancing spatial information processing while preserving anatomical continuity. Lastly, to effectively align the localization lens representations with textual features, we incorporate decoupled contrastive loss (DCL) alongside the standard loss function. This ensures better feature discrimination and robustness, particularly in data limited medical settings. Through extensive evaluations on medical visual question answering (Med-VQA) datasets, we show that our methodology improves localization-driven performance across different Med-VLM architectures. Our analysis of localization-based questions further reveals that improvements in anatomy and spatial reasoning directly enhance the overall accuracy of Med-VQA upto 6.2%. The proposed approach is model-agnostic and can be seamlessly integrated into existing Med-VLM pipelines. The dataset, code, and trained models will be made publicly available at https://github.com/CVLABLUMS/localizationlens.

Problem

• Medical Vision-Language Models
• Anatomical understanding
• Spatial positioning
• Medical reasoning
• Med-VQA

Innovation

• Medical Vision-Language Models
• Localization Lens
• Pixel Shuffle
• Decoupled Contrastive Loss
• Medical Visual Question Answering