🤖 AI Summary
Medical radiology reports exhibit linguistic complexity, imbalanced entity distributions, and insufficient cross-modal visual–linguistic contextual modeling. To address these challenges, this work proposes an entity-centric, context-aware vision–language pretraining framework. Its key contributions are: (1) a novel entity-centered context distillation mechanism that leverages large language models to extract fine-grained semantic representations of medical entities; (2) a joint pretraining objective combining entity-aware masked language modeling with context-guided medical image super-resolution; and (3) a multi-scale cross-modal feature fusion architecture that enhances image–text alignment and reconstruction fidelity. Evaluated on multiple public medical imaging benchmarks, the proposed method consistently outperforms state-of-the-art approaches across diverse downstream tasks—including classification, segmentation, and detection—and establishes new performance records on several standard metrics.
📝 Abstract
Despite significant advancements in medical vision-language pre-training, existing methods have largely overlooked the inherent entity-specific context within radiology reports and the complex cross-modality contextual relationships between text and images. To close this gap, we propose a novel Entity-centered Context-aware Medical Vision-language Pre-training (ECAMP) framework, which is designed to enable a more entity-centered and context-sensitive interpretation of medical data. Utilizing the recent powerful large language model, we distill entity-centered context from medical reports, which enables ECAMP to gain more effective supervision from the text modality. By further pre-training our model with carefully designed entity-aware, context-enhanced masked language modeling and context-guided super-resolution tasks, ECAMP significantly refines the interplay between text and image modalities, leading to an enhanced ability to extract entity-centered contextual features. Besides, our proposed multi-scale context fusion design also improves the semantic integration of both coarse and fine-level image representations, prompting better performance for multi-scale downstream applications. Combining these components leads to significant performance leaps over current state-of-the-art methods and establishes a new standard for cross-modality learning in medical imaging, whose effectiveness is demonstrated by our extensive experiments on various tasks including classification, segmentation, and detection across several public datasets. Code and models are available at https://github.com/ToniChopp/ECAMP.
