CLIP Architecture for Abdominal CT Image-Text Alignment and Zero-Shot Learning: Investigating Batch Composition and Data Scaling

2026-04-15

AI Summary

This work investigates the impact of training batch composition on vision–language representation learning in 3D medical imaging, specifically focusing on aligning abdominal CT scans with radiology reports. Leveraging the Merlin dual-encoder architecture and symmetric InfoNCE loss, the study systematically analyzes how the ratio of normal to abnormal samples and dataset scale influence zero-shot performance. Notably, it reveals— for the first time in 3D medical imaging—that implicit diversity from random sampling consistently outperforms explicit class-balancing strategies, challenging conventional assumptions about data balancing. Experimental results demonstrate that the proposed approach achieves a zero-shot macro F1 score of 74.45% across 30 pathological findings; explicit balancing uniformly degrades performance by 2.4–2.8 points, while increasing data scale yields only sublinear gains.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Vision-language models trained with contrastive learning on paired medical images and reports show strong zero-shot diagnostic capabilities, yet the effect of training batch composition on learned representations remains unexplored for 3D medical imaging. We reproduce Merlin, a dual-encoder model that aligns 3D abdominal CT volumes with radiology reports using symmetric InfoNCE loss, achieving a zero-shot macro F1 of 74.45% across 30 findings (original: 73.00%). We then investigate two axes of variation. First, we control the normal-to-abnormal ratio within training batches at 25:75, 50:50, and 75:25 using section-level balanced sampling on the full dataset. All three configurations underperform the unbalanced baseline by 2.4 to 2.8 points, with 75:25 achieving the best result (72.02%) among balanced variants. Second, we conduct data scaling ablations on a 4,362-study subset, training with 20%, 40%, and 100% of the data. Performance scales sub-linearly from 65.26% to 71.88%, with individual findings varying dramatically in data sensitivity. Enforcing 50:50 balanced sampling on the same subset further degrades performance to 68.01%, confirming that explicit class balancing hurts regardless of dataset or balancing granularity. Our results indicate that the stochastic diversity of random sampling, combined with Merlin's alternating batching over anatomical subsections, provides more effective regularization than engineered class ratios at the small batch sizes required by 3D medical volumes.

Problem

• batch composition
• 3D medical imaging
• zero-shot learning
• class balancing
• data scaling

Innovation

• CLIP architecture
• zero-shot learning
• batch composition
• data scaling
• 3D medical imaging