GBT-SAM: A Parameter-Efficient Depth-Aware Model for Generalizable Brain tumour Segmentation on mp-MRI

2025-03-06

AI Summary

To address three key challenges in multi-parametric MRI brain tumor segmentation—poor cross-disease generalization (across adult/child glioma, meningioma, and sub-Saharan glioma), high computational cost, and insufficient multimodal information integration—this work pioneers the adaptation of Segment Anything Model (SAM) to the brain tumor domain. We propose a depth-conditioned module to explicitly model inter-slice feature correlations and integrate it with Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning—requiring only 6.5M trainable parameters. Our ViT-based two-stage training first refines full-modality patch embeddings, then injects LoRA adapters and the depth-conditioned module to fuse T1, T2, and FLAIR sequences. On the adult glioma dataset, our method achieves a Dice score of 93.54% and sets a new state-of-the-art for cross-disease generalization. The code and models are publicly released.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Gliomas are brain tumours that stand out for their highly lethal and aggressive nature, which demands a precise approach in their diagnosis. Medical image segmentation plays a crucial role in the evaluation and follow-up of these tumours, allowing specialists to analyse their morphology. However, existing methods for automatic glioma segmentation often lack generalization capability across other brain tumour domains, require extensive computational resources, or fail to fully utilize the multi-parametric MRI (mp-MRI) data used to delineate them. In this work, we introduce GBT-SAM, a novel Generalizable Brain Tumour (GBT) framework that extends the Segment Anything Model (SAM) to brain tumour segmentation tasks. Our method employs a two-step training protocol: first, fine-tuning the patch embedding layer to process the entire mp-MRI modalities, and second, incorporating parameter-efficient LoRA blocks and a Depth-Condition block into the Vision Transformer (ViT) to capture inter-slice correlations. GBT-SAM achieves state-of-the-art performance on the Adult Glioma dataset (Dice Score of $93.54$) while demonstrating robust generalization across Meningioma, Pediatric Glioma, and Sub-Saharan Glioma datasets. Furthermore, GBT-SAM uses less than 6.5M trainable parameters, thus offering an efficient solution for brain tumour segmentation. \ Our code and models are available at https://github.com/vpulab/med-sam-brain .

Problem

• Improves generalization in brain tumour segmentation across domains.
• Reduces computational resources for multi-parametric MRI data processing.
• Enhances segmentation accuracy using parameter-efficient depth-aware techniques.

Innovation

• Extends Segment Anything Model for brain tumours
• Uses parameter-efficient LoRA blocks in ViT
• Incorporates Depth-Condition block for inter-slice correlations