🤖 AI Summary
To address the scarcity of high-quality annotations and low model efficiency in 3D medical image segmentation, this work pioneers the zero-shot cross-modal transfer of the video foundation model SAM 2 to 3D medical imaging—treating consecutive 2D slices as video frames and enabling fully automatic 3D segmentation from a single-frame interaction. We propose a temporal-propagation-based 3D slice sequence modeling strategy and a plug-and-play interactive prompting fine-tuning pipeline (supporting point or bounding-box inputs), drastically reducing clinicians’ annotation burden. Evaluated on multi-center BraTS2020 and MSD datasets, our method achieves a Dice score of 0.72 for brain tumor subregions—comparable to fully supervised baselines—and completes whole-volume segmentation with only one user interaction, improving annotation efficiency by over 10×. This study establishes the first zero-shot cross-modal transfer paradigm for video-pretrained models in 3D medical segmentation.
📝 Abstract
Interactive medical image segmentation (IMIS) has shown significant potential in enhancing segmentation accuracy by integrating iterative feedback from medical professionals. However, the limited availability of enough 3D medical data restricts the generalization and robustness of most IMIS methods. The Segment Anything Model (SAM), though effective for 2D images, requires expensive semi-auto slice-by-slice annotations for 3D medical images. In this paper, we explore the zero-shot capabilities of SAM 2, the next-generation Meta SAM model trained on videos, for 3D medical image segmentation. By treating sequential 2D slices of 3D images as video frames, SAM 2 can fully automatically propagate annotations from a single frame to the entire 3D volume. We propose a practical pipeline for using SAM 2 in 3D medical image segmentation and present key findings highlighting its efficiency and potential for further optimization. Concretely, numerical experiments on the BraTS2020 and the medical segmentation decathlon datasets demonstrate that SAM 2 still has a gap with supervised methods but can narrow the gap in specific settings and organ types, significantly reducing the annotation burden on medical professionals. Our code will be open-sourced and available at https://github.com/Chuyun-Shen/SAM_2_Medical_3D.