Test-Time Modality Generalization for Medical Image Segmentation

2025-02-27

AI Summary

This paper addresses the generalization bottleneck of medical image segmentation across unseen imaging modalities (e.g., colonoscopy, ultrasound, dermatoscopy, radiography) by proposing a novel test-time modality generalization (TTMG) paradigm. Methodologically, it introduces (1) modality-aware style projection (MASP) for dynamic modality alignment, and (2) modality-sensitive instance whitening (MSIW) to suppress modality-specific interference, enhanced by modality likelihood estimation and feature covariance regularization for improved robustness. Crucially, the approach requires no source-domain modality labels and supports open-set modality adaptation. Extensive evaluation across 11 cross-modality benchmarks demonstrates substantial performance gains over state-of-the-art domain generalization and adaptation methods. To our knowledge, this is the first framework enabling truly plug-and-play segmentation generalization to arbitrary unseen modalities without retraining or modality-specific supervision.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Data transparency and provenance
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Generalizable medical image segmentation is essential for ensuring consistent performance across diverse unseen clinical settings. However, existing methods often overlook the capability to generalize effectively across arbitrary unseen modalities. In this paper, we introduce a novel Test-Time Modality Generalization (TTMG) framework, which comprises two core components: Modality-Aware Style Projection (MASP) and Modality-Sensitive Instance Whitening (MSIW), designed to enhance generalization in arbitrary unseen modality datasets. The MASP estimates the likelihood of a test instance belonging to each seen modality and maps it onto a distribution using modality-specific style bases, guiding its projection effectively. Furthermore, as high feature covariance hinders generalization to unseen modalities, the MSIW is applied during training to selectively suppress modality-sensitive information while retaining modality-invariant features. By integrating MASP and MSIW, the TTMG framework demonstrates robust generalization capabilities for medical image segmentation in unseen modalities a challenge that current methods have largely neglected. We evaluated TTMG alongside other domain generalization techniques across eleven datasets spanning four modalities (colonoscopy, ultrasound, dermoscopy, and radiology), consistently achieving superior segmentation performance across various modality combinations.

Problem

• Enhances generalization across unseen medical modalities
• Improves segmentation accuracy in diverse clinical settings
• Addresses modality-sensitive feature suppression during training

Innovation

• Test-Time Modality Generalization
• Modality-Aware Style Projection
• Modality-Sensitive Instance Whitening