LocAnyMed: Vision-Language Grounding for Multimodal Medical Images

2026-08-04

AI Summary

This study addresses the fragmented landscape of medical visual grounding—characterized by disparate modalities, datasets, and task formulations—and establishes the first unified benchmark for multimodal medical image localization. The authors introduce LocAnyMed-200K, a large-scale dataset comprising 200,000 samples spanning CT, X-ray, ultrasound, and optical imaging modalities, accompanied by free-form instructions that support negative samples, point coordinates, and multiple bounding boxes. To enhance interpretability, they further curate LocAnyMed-CoT-20K, a 20,000-sample subset featuring chain-of-thought reasoning annotations. Through full-parameter fine-tuning of the LocateAnything-3B model with extensive domain-specific supervision, the proposed approach achieves a substantial improvement in performance, raising the F1@IoU 0.50 score from 10.64 to 85.59 on an independent test set, thereby providing a unified, accurate, and interpretable foundation for multimodal medical visual grounding.

Abstract

Medical visual grounding connects free-form clinical queries to spatial evidence in medical images and is an important component of interpretable medical artificial intelligence. However, general-purpose grounding models are predominantly trained on natural images, while existing medical localization resources remain fragmented across imaging modalities, datasets, and task formulations. To address this gap, we construct LocAnyMed-200K, a multimodal medical visual grounding dataset containing approximately 200K image-query-answer examples across computed tomography, optical medical imaging, ultrasound, and X-ray. We harmonize heterogeneous detection and localization resources into a unified free-form instruction format that supports one or multiple bounding boxes, point coordinates, and no-target outputs for negative queries. Full-parameter fine-tuning of LocateAnything-3B on LocAnyMed-200K improves F1@IoU 0.50 from 10.64 to 85.59 on a held-out evaluation split, demonstrating that large-scale domain-specific supervision can equip a general grounding model with effective medical localization capabilities. Beyond spatial coordinates, a clinically interpretable grounding system should also communicate the evidence supporting its prediction. We therefore derive LocAnyMed-CoT-20K, a rationale-augmented subset that connects anatomical context, visual observations, and spatial conclusions through structured reasoning and further improves cross-source generalization through fine-tuning. Together, these resources provide a unified foundation for studying both localization accuracy and rationale quality across heterogeneous medical imaging modalities. The code is publicly available at https://github.com/MiliLab/LocAnyMed.

Problem

• medical visual grounding
• multimodal medical images
• domain-specific supervision
• heterogeneous imaging modalities
• interpretable AI

Innovation

• medical visual grounding
• multimodal medical imaging
• instruction tuning
• chain-of-thought reasoning
• domain-specific foundation model