Parameter-Efficient Adaptation of SAM3 for Prompt-Driven Surgical Concept Segmentation

2026-07-26

AI Summary

This work addresses the performance degradation of SAM3 in surgical image segmentation due to domain shift and the high computational cost of existing medical adaptation methods that require full-parameter fine-tuning. To overcome these limitations, we introduce low-rank adaptation (LoRA) into SAM3 for the first time, freezing the vision backbone and fine-tuning only 0.98% of the parameters in the prompt encoder, detector, and tracker. The proposed approach significantly outperforms zero-shot SAM3 and state-of-the-art baselines across multiple surgical segmentation tasks, enables efficient training on a single consumer-grade GPU, and produces segmentation outputs directly usable for robotic surgery scene reconstruction and physical simulation, thereby achieving parameter-efficient, accurate, and clinically practical surgical concept segmentation.

Abstract

Efficient surgical segmentation empowers clinical diagnosis, intraoperative monitoring, and downstream robotic pipelines for reconstruction and simulation. Although prompt-driven foundation models like Segment Anything Model 3 (SAM3) achieve strong segmentation performance on natural images, surgical data exhibits domain gaps against its pre-training data, resulting in degraded segmentation accuracy. Furthermore, existing medical SAM methods require full-parameter fine-tuning, incurring heavy computational consumption and low efficiency. To address these limitations, this work proposes a parameter-efficient Low-Rank Adaptation (LoRA) adaptation of SAM3 for surgical concept segmentation. We inject low-rank adapters into the prompt encoder, detector and tracker while fully freezing the vision backbone, which only optimizes 0.98% of the total model parameters and supports training on a single consumer GPU. Comprehensive experiments demonstrate that our method consistently outperforms zero-shot SAM3 and other mainstream baselines, and the generated segmentation results can be directly deployed to support downstream robotic surgical scene reconstruction and physical simulation pipelines.

Problem

• surgical segmentation
• domain gap
• parameter-efficient adaptation
• foundation models
• computational efficiency

Innovation

• Low-Rank Adaptation
• SAM3
• Surgical Segmentation
• Parameter-Efficient Tuning
• Prompt-Driven