Semi-Supervised 3D Medical Segmentation from 2D Natural Images Pretrained Model

2025-09-18

AI Summary

To address the scarcity of annotated data in semi-supervised 3D medical image segmentation, this paper proposes a cross-dimensional knowledge transfer framework that effectively adapts 2D general-purpose vision models—pretrained on natural images—to 3D medical segmentation tasks. Methodologically, we design a model-agnostic progressive co-training scheme integrating pseudo-label generation, iterative self-training, and knowledge distillation, augmented by a learning-rate-guided adaptive sampling strategy to suppress pseudo-label noise. Our contributions are twofold: (i) the first dimension-aligned knowledge distillation from 2D foundation models to 3D medical segmentation, enabling effective cross-dimensional transfer; and (ii) a novel co-training paradigm requiring no architectural modifications, significantly enhancing generalizability and structural compatibility. Extensive experiments demonstrate state-of-the-art performance across multiple public benchmarks, outperforming 13 existing methods with superior accuracy and robustness.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Semi-Supervised Learning
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

This paper explores the transfer of knowledge from general vision models pretrained on 2D natural images to improve 3D medical image segmentation. We focus on the semi-supervised setting, where only a few labeled 3D medical images are available, along with a large set of unlabeled images. To tackle this, we propose a model-agnostic framework that progressively distills knowledge from a 2D pretrained model to a 3D segmentation model trained from scratch. Our approach, M&N, involves iterative co-training of the two models using pseudo-masks generated by each other, along with our proposed learning rate guided sampling that adaptively adjusts the proportion of labeled and unlabeled data in each training batch to align with the models' prediction accuracy and stability, minimizing the adverse effect caused by inaccurate pseudo-masks. Extensive experiments on multiple publicly available datasets demonstrate that M&N achieves state-of-the-art performance, outperforming thirteen existing semi-supervised segmentation approaches under all different settings. Importantly, ablation studies show that M&N remains model-agnostic, allowing seamless integration with different architectures. This ensures its adaptability as more advanced models emerge. The code is available at https://github.com/pakheiyeung/M-N.

Problem

• Transferring 2D natural image knowledge to 3D medical segmentation
• Addressing limited labeled 3D medical images with semi-supervised learning
• Minimizing adverse effects from inaccurate pseudo-masks during training

Innovation

• Knowledge distillation from 2D to 3D model
• Iterative co-training with pseudo-masks
• Adaptive sampling based on learning rates