Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation

2026-07-20

AI Summary

This work addresses the performance degradation in medical image segmentation under distribution shifts during inference by proposing a training-free test-time adaptation framework. The method uniquely integrates a memory mechanism, semantic priors, and cross-image structural alignment. It employs a noise-aware module to dynamically select reliable text-image predictions for constructing an online memory bank and introduces a correlation-driven prototype alignment module to enable cross-modal prediction filtering and enforce structural consistency. Without updating model parameters, the approach effectively prevents degradation of pretrained features and substantially enhances segmentation robustness. Extensive experiments demonstrate that the proposed method significantly outperforms existing approaches across multiple medical segmentation benchmarks, achieving improvements of up to 12.2% in Dice Similarity Coefficient (DSC) and 11.7% in mean Intersection over Union (mIoU).

Abstract

Test-time adaptation (TTA) aims to mitigate distribution shifts by adapting models with unlabeled target data at inference time. While TTA with vision-language models (VLMs) has shown promising results in classification, extending it to medical image segmentation remains challenging. In this setting, the adaptation gains from optimizing on VLM-generated predictions are often outweighed by the degradation to the VLM's strong pretrained features caused by noisy, update-driven learning, resulting in limited and unstable improvements. We therefore propose Memory-Supported Synergistic Adaptation (MSSA), a novel training-free TTA framework for medical image segmentation. Without updating model parameters, MSSA dynamically selects reliable image-text predictions to construct an online memory, uses them as text-guided semantic priors, and couples them with cross-image structural alignment for robust adaptation. Specifically, MSSA consists of (i) a noise-aware memory construction module that filters and stabilizes cross-modal predictions, and (ii) a relevance-driven prototype alignment module that aligns the target sample with structurally consistent memory samples and their reliable predictions to improve adaptation. Extensive experiments on multiple medical segmentation benchmarks demonstrate that MSSA consistently improves VLM-based segmentation models and outperforms existing fine-tuning-based TTA methods by a clear margin, with gains of up to 12.2% DSC and 11.7% mIoU. Project page: https://lingrayy.github.io/MSSA/ .

Problem

• test-time adaptation
• medical image segmentation
• vision-language models
• distribution shift
• training-free

Innovation

• Test-Time Adaptation
• Vision-Language Models
• Medical Image Segmentation
• Memory-Based Learning
• Training-Free Adaptation