Evaluating Pre-trained Convolutional Neural Networks and Foundation Models as Feature Extractors for Content-based Medical Image Retrieval

2024-09-14

Engineering applications of artificial intelligence

AI Summary

This work systematically evaluates the transferability of pretrained CNNs (e.g., ResNet) versus vision foundation models (ViTs, DINOv2, SAM, CLIP) as feature extractors for cross-modal, few-shot medical image retrieval (CBMIR), while analyzing the impact of input image resolution. We propose a medical-image-specific feature normalization and metric fusion strategy. To our knowledge, this is the first unified benchmark evaluation of foundation models—including SAM and DINOv2—in CBMIR. Experiments show that DINOv2 achieves R@10 = 72.3% on the RSNA CXR dataset without fine-tuning, substantially outperforming conventional CNNs; enables effective cross-domain retrieval; and accelerates inference by 3.2×. t-SNE visualization and linear probing further confirm its robustness and strong transferability.

Technology Category

Computer Vision: Multi-modal VisionMachine Learning: Deep Neural Architectures and Foundation ModelsData Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAGWeb Mining and Content Analysis: Large pretrained models with web dataGraph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Problem

• Evaluating pre-trained CNNs and foundation models for medical image retrieval
• Investigating performance on 2D and 3D medical images across different sizes
• Comparing CNN and foundation model effectiveness in content-based retrieval tasks

Innovation

• Pre-trained CNNs and foundation models for feature extraction
• Evaluated performance on 2D and 3D medical images
• Investigated image size impact on retrieval performance