LesionLocator: Zero-Shot Universal Tumor Segmentation and Tracking in 3D Whole-Body Imaging

2025-02-28

AI Summary

This work addresses the challenge of zero-shot universal tumor segmentation and longitudinal tracking in 3D whole-body medical imaging, proposing the first end-to-end 4D promptable model. Methodologically: (1) we construct and publicly release a large-scale synthetic 4D longitudinal medical dataset; (2) we design a dense spatial prompting mechanism coupled with a zero-shot transfer architecture, enabling localization, segmentation, and cross-temporal tracking of arbitrary tumor types within whole-body volumetric scans. Contributions and results: our approach overcomes key bottlenecks in zero-shot generalization—trained jointly on real clinical scans (23,262 annotated cases) and multi-disease synthetic data, it achieves segmentation Dice scores ~10 points higher than baseline methods, reaching expert-level performance; longitudinal tracking accuracy attains state-of-the-art. Both the model and dataset are fully open-sourced to advance general-purpose medical image analysis.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: 3D Computer Vision
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

In this work, we present LesionLocator, a framework for zero-shot longitudinal lesion tracking and segmentation in 3D medical imaging, establishing the first end-to-end model capable of 4D tracking with dense spatial prompts. Our model leverages an extensive dataset of 23,262 annotated medical scans, as well as synthesized longitudinal data across diverse lesion types. The diversity and scale of our dataset significantly enhances model generalizability to real-world medical imaging challenges and addresses key limitations in longitudinal data availability. LesionLocator outperforms all existing promptable models in lesion segmentation by nearly 10 dice points, reaching human-level performance, and achieves state-of-the-art results in lesion tracking, with superior lesion retrieval and segmentation accuracy. LesionLocator not only sets a new benchmark in universal promptable lesion segmentation and automated longitudinal lesion tracking but also provides the first open-access solution of its kind, releasing our synthetic 4D dataset and model to the community, empowering future advancements in medical imaging. Code is available at: www.github.com/MIC-DKFZ/LesionLocator

Problem

• Develops zero-shot lesion tracking and segmentation in 3D imaging.
• Addresses limitations in longitudinal medical data availability.
• Enhances generalizability and accuracy in lesion segmentation and tracking.

Innovation

• Zero-shot lesion tracking and segmentation
• End-to-end 4D tracking with dense prompts
• Open-access synthetic 4D dataset release