Location-Aware Fine-Grained Representation Learning for Medical Vision Foundation Models

2026-08-01

AI Summary

Current medical visual encoders struggle to simultaneously capture fine-grained clinical semantics and spatial consistency, limiting the spatial grounding capabilities of large vision-language models (LVLMs). This work proposes LoFi, a novel approach that introduces bidirectional image-text grounding into medical vision foundation model training for the first time. By jointly optimizing both the localization and the grounded textual descriptions, LoFi leverages a lightweight large language model to guide the visual encoder in learning position-aware, fine-grained representations—without requiring explicit region-level supervision. To support this framework, the authors construct MedG, a large-scale medical grounding dataset, and integrate self-supervised with weakly supervised learning strategies. Experiments demonstrate that LoFi significantly outperforms both general-purpose and medical-specific vision foundation models, as well as state-of-the-art LVLMs, on tasks including phrase grounding, visual question answering, and organ classification under perturbation.

Abstract

Fine-grained visual representations are essential for medical image analysis, particularly when diagnostically relevant evidence is subtle and spatially localized. Modern transformer-based medical vision encoders must therefore learn patch-level representations that are both clinically meaningful and spatially consistent. Without these properties, large vision-language models (LVLMs) operate on an ambiguous visual foundation, limiting their ability to generate clinically reliable and spatially grounded responses. However, existing training strategies for medical vision encoders rarely achieve both objectives. Image-text alignment provides clinically meaningful supervision primarily at the image level, leaving the spatial localization of diagnostic evidence weakly constrained. In contrast, self-supervised learning promotes spatial consistency but lacks the semantic supervision needed to distinguish visually similar yet clinically distinct regions. To address this gap, we present LoFi, a medical vision foundation model built on location-aware fine-grained representation learning. LoFi trains a vision encoder with a lightweight large language model under grounding and grounded captioning objectives. Because these objectives require predicting location from clinical text and vice versa, spatial consistency emerges without any explicit patch-level regularization. To enable training at scale, we construct MedG, a large-scale medical grounding dataset of 4.48M image-text-box triplets curated from 84 datasets spanning 7 modalities. Across phrase grounding, visual question answering, and region-based organ classification under perturbations, LoFi consistently outperforms general-purpose and medical vision foundation models as well as state-of-the-art LVLMs. Code is available at https://github.com/myeongkyunkang/lofi-medg.

Problem

• fine-grained representation
• spatial consistency
• clinical semantics
• medical vision foundation models
• location-aware learning

Innovation

• location-aware representation
• fine-grained learning
• medical vision foundation model
• grounded captioning
• vision-language grounding