TopoLoRA-SAM: Topology-Aware Parameter-Efficient Adaptation of Foundation Segmenters for Thin-Structure and Cross-Domain Binary Semantic Segmentation

2026-01-05

arXiv.org

AI Summary

This work addresses the challenges faced by foundational segmentation models like SAM in fine-structured and cross-modal binary semantic segmentation tasks, where full fine-tuning is computationally expensive and prone to catastrophic forgetting. To overcome these limitations, we propose TopoLoRA-SAM—the first framework integrating topological awareness with parameter-efficient fine-tuning. Our approach freezes the ViT encoder and injects low-rank adaptation (LoRA) modules alongside lightweight spatial convolutional adapters, optionally augmented with differentiable clDice-based topological supervision. Training only 5.2% of the model parameters (approximately 4.9M), TopoLoRA-SAM achieves state-of-the-art average Dice scores across five benchmark datasets, with particularly significant improvements in segmentation accuracy and robustness on CHASE_DB1, outperforming fully fine-tuned task-specific models.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Large Multimodal Models (LMMs)
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Foundation segmentation models such as the Segment Anything Model (SAM) exhibit strong zero-shot generalization through large-scale pretraining, but adapting them to domain-specific semantic segmentation remains challenging, particularly for thin structures (e.g., retinal vessels) and noisy modalities (e.g., SAR imagery). Full fine-tuning is computationally expensive and risks catastrophic forgetting. We propose \textbf{TopoLoRA-SAM}, a topology-aware and parameter-efficient adaptation framework for binary semantic segmentation. TopoLoRA-SAM injects Low-Rank Adaptation (LoRA) into the frozen ViT encoder, augmented with a lightweight spatial convolutional adapter and optional topology-aware supervision via differentiable clDice. We evaluate our approach on five benchmarks spanning retinal vessel segmentation (DRIVE, STARE, CHASE\_DB1), polyp segmentation (Kvasir-SEG), and SAR sea/land segmentation (SL-SSDD), comparing against U-Net, DeepLabV3+, SegFormer, and Mask2Former. TopoLoRA-SAM achieves the best retina-average Dice and the best overall average Dice across datasets, while training only \textbf{5.2\%} of model parameters ($\sim$4.9M). On the challenging CHASE\_DB1 dataset, our method substantially improves segmentation accuracy and robustness, demonstrating that topology-aware parameter-efficient adaptation can match or exceed fully fine-tuned specialist models. Code is available at : https://github.com/salimkhazem/Seglab.git

Problem

• foundation segmenters
• thin-structure segmentation
• cross-domain segmentation
• parameter-efficient adaptation
• binary semantic segmentation

Innovation

• TopoLoRA-SAM
• parameter-efficient adaptation
• topology-aware segmentation
• Low-Rank Adaptation (LoRA)
• binary semantic segmentation