Boosting Medical Vision-Language Pretraining via Momentum Self-Distillation under Limited Computing Resources

2025-12-02

AI Summary

Addressing the dual challenges of scarce labeled data and high computational cost in contrastive learning (CL) for medical imaging, this paper proposes a low-resource-efficient vision-language pretraining framework. Methodologically, it introduces (1) a momentum-based self-distillation mechanism to strengthen cross-modal semantic alignment and knowledge transfer, and (2) gradient accumulation to emulate large-batch training on a single GPU, thereby improving model convergence and generalization. Evaluated on medical benchmarks, the approach achieves state-of-the-art performance across zero-shot classification, few-shot adaptation (AUC-ROC > 90%), and cross-modal retrieval (2–3% improvement in recall@k), while significantly reducing hardware requirements and training cost. The framework establishes a scalable, resource-efficient paradigm for multimodal medical modeling under constrained computational and annotation budgets.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

In medical healthcare, obtaining detailed annotations is challenging, highlighting the need for robust Vision-Language Models (VLMs). Pretrained VLMs enable fine-tuning on small datasets or zero-shot inference, achieving performance comparable to task-specific models. Contrastive learning (CL) is a key paradigm for training VLMs but inherently requires large batch sizes for effective learning, making it computationally demanding and often limited to well-resourced institutions. Moreover, with limited data in healthcare, it is important to prioritize knowledge extraction from both data and models during training to improve performance. Therefore, we focus on leveraging the momentum method combined with distillation to simultaneously address computational efficiency and knowledge exploitation. Our contributions can be summarized as follows: (1) leveraging momentum self-distillation to enhance multimodal learning, and (2) integrating momentum mechanisms with gradient accumulation to enlarge the effective batch size without increasing resource consumption. Our method attains competitive performance with state-of-the-art (SOTA) approaches in zero-shot classification, while providing a substantial boost in the few-shot adaption, achieving over 90% AUC-ROC and improving retrieval tasks by 2-3%. Importantly, our method achieves high training efficiency with a single GPU while maintaining reasonable training time. Our approach aims to advance efficient multimodal learning by reducing resource requirements while improving performance over SOTA methods. The implementation of our method is available at https://github.com/phphuc612/MSD .

Problem

• Enhances medical vision-language models with limited computing resources
• Addresses large batch size demands in contrastive learning efficiently
• Improves knowledge extraction from data and models for better performance

Innovation

• Momentum self-distillation enhances multimodal learning efficiently
• Gradient accumulation increases effective batch size without extra resources
• Achieves high performance on single GPU with limited computing