SAM+D: Parameter-Efficient Dimensional Lifting of SAM-Family Models via Depth-Routed LoRA and Depth Shifting

2026-07-31

AI Summary

This work addresses the challenge of extending 2D foundation models like SAM to 3D/4D volumetric data, where existing approaches either neglect inter-slice context or require extensive architectural modifications and retraining. The authors propose a parameter-efficient dimensionality lifting framework that freezes the original SAM parameters and introduces Depth-Routed LoRA (DRLoRA) for spatially adaptive low-rank updates, complemented by a parameter-free Depth Shift Module (DSM) to model cross-slice feature interactions. Requiring fine-tuning of only 2.8%–3.7% of parameters, the method enables end-to-end 3D/4D spatiotemporal segmentation. Evaluated on 3D medical imaging benchmarks such as KiTS and the 4D cell tracking task Fluo-N3DH-SIM+, it achieves competitive or superior performance under single-point prompting while significantly reducing training overhead.

Abstract

Existing methods for adapting 2D foundation models such as SAM to 3D volumes either process slices independently---ignoring inter-slice context---or require substantial architectural changes and retraining. In this paper, we present \textbf{SAM+D}, a parameter-efficient framework that lifts SAM-family models by one spatial dimension---enabling 3D volumetric segmentation from 2D SAM and, for the first time via parameter-efficient fine-tuning, end-to-end 4D (3D+T) spatiotemporal segmentation from video-based SAM2---while keeping the vast majority of pre-trained parameters frozen. SAM+D introduces two lightweight, model-agnostic modules into frozen transformer blocks: (1)~\textbf{Depth-Routed LoRA (DRLoRA)} experts with learned routing for spatially adaptive low-rank updates, and (2)~\textbf{Depth Shift Modules (DSM)} for cross-slice feature exchange at zero additional parameter cost. Together, they provide volume-level context while tuning only ${\sim}$2.8\% of parameters for SAM and ${\sim}$3.7\% for SAM2. We evaluate SAM+D in two distinct settings, each lifting the base model by one spatial dimension: 3D segmentation, where SAM(2D$\,\to\,$3D) is evaluated on four CT benchmarks (KiTS, Pancreas, LiTS, Colon), and 4D segmentation, where SAM2 (2D+T$\,\to\,$3D+T) is evaluated on a cell tracking challenge (CTC) dataset (Fluo-N3DH-SIM+). In both settings SAM+D achieves competitive or superior results under the single-point prompt setting while using fewer trainable parameters than existing methods, demonstrating that SAM+D generalizes across SAM-family architectures, target dimensionalities (3D, 4D), and domains spanning medical imaging and bio-scene understanding. Code is publicly available at https://github.com/JerrySongCST/SAM-Plus-D.

Problem

• dimensional lifting
• 3D segmentation
• 4D segmentation
• parameter-efficient adaptation
• SAM-family models

Innovation

• Depth-Routed LoRA
• Depth Shift Modules
• Parameter-Efficient Fine-Tuning
• Dimensional Lifting
• SAM-family Models