RadiomicsRetrieval: A Customizable Framework for Medical Image Retrieval Using Radiomics Features

2025-07-11

AI Summary

Existing medical image retrieval methods are predominantly limited to 2D images and rely on fully annotated queries, resulting in insufficient clinical flexibility. To address this, we propose a customizable 3D medical image retrieval framework that— for the first time—integrates radiomic features with promptable segmentation models (e.g., SAM), incorporates anatomical position embeddings (APE) to enhance spatial contextual modeling, and employs contrastive learning to align handcrafted tumor descriptors with deep semantic embeddings. Our method enables retrieval based on minimal user prompts (e.g., points or bounding boxes), supporting shape-, location-, or partial-feature–driven queries, thereby significantly improving retrieval specificity and lesion localization accuracy. Evaluated on public lung CT and brain MRI datasets, our approach outperforms state-of-the-art methods in both retrieval precision and clinical utility, demonstrating strong suitability for decision support in large-scale clinical imaging repositories.

Technology Category

• Computer Vision: Image and Video Retrieval
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Medical image retrieval is a valuable field for supporting clinical decision-making, yet current methods primarily support 2D images and require fully annotated queries, limiting clinical flexibility. To address this, we propose RadiomicsRetrieval, a 3D content-based retrieval framework bridging handcrafted radiomics descriptors with deep learning-based embeddings at the tumor level. Unlike existing 2D approaches, RadiomicsRetrieval fully exploits volumetric data to leverage richer spatial context in medical images. We employ a promptable segmentation model (e.g., SAM) to derive tumor-specific image embeddings, which are aligned with radiomics features extracted from the same tumor via contrastive learning. These representations are further enriched by anatomical positional embedding (APE). As a result, RadiomicsRetrieval enables flexible querying based on shape, location, or partial feature sets. Extensive experiments on both lung CT and brain MRI public datasets demonstrate that radiomics features significantly enhance retrieval specificity, while APE provides global anatomical context essential for location-based searches. Notably, our framework requires only minimal user prompts (e.g., a single point), minimizing segmentation overhead and supporting diverse clinical scenarios. The capability to query using either image embeddings or selected radiomics attributes highlights its adaptability, potentially benefiting diagnosis, treatment planning, and research on large-scale medical imaging repositories. Our code is available at https://github.com/nainye/RadiomicsRetrieval.

Problem

• Enables 3D medical image retrieval using radiomics and deep learning
• Supports flexible queries with minimal user prompts for clinical use
• Enhances retrieval specificity with tumor-level features and anatomical context

Innovation

• 3D content-based retrieval using radiomics and deep learning
• Promptable segmentation model for tumor-specific embeddings
• Anatomical positional embedding enriches global context