DistMedVL: Distributional Vision-Language Alignment for Uncertainty-Aware Medical Image Segmentation

2026-08-06

AI Summary

This work addresses the performance degradation of existing medical image segmentation methods under domain shift, which stems from their neglect of uncertainty in visual and textual representations during cross-modal alignment. To overcome this limitation, we propose DistMedVL, a probabilistic vision-language framework that introduces a lightweight Probabilistic Cross-Modal Adapter (PCM-Adapter) on top of frozen encoders. DistMedVL is the first to incorporate Gaussian distribution modeling and Mahalanobis distance matching for medical vision–language alignment. Through a Mahalanobis Alignment Module (MAM) and a Distribution Flow Module (DFM), our approach enables cross-modal confidence estimation and vision-guided refinement of textual distributions, adding only 6.3 million trainable parameters. Evaluated on eight medical segmentation benchmarks, DistMedVL significantly outperforms current state-of-the-art methods, demonstrating superior data efficiency, robustness to perturbations, and cross-dataset generalization capability.

Abstract

Cross-modal alignment of visual and textual representations is fundamental to multimodal medical image understanding, yet remains hindered by uncertainty in both modalities under real-world clinical conditions. Existing vision-language segmentation methods rely on deterministic cross-modal matching, which overlooks aleatoric uncertainty from ambiguous boundaries and epistemic uncertainty from limited training data, leading to fragile performance under domain shift. To address this issue, we propose DistMedVL, a probabilistic vision-language framework that introduces a lightweight Probabilistic Cross-Modal Adapter (PCM-Adapter) upon frozen encoders to explicitly model representational uncertainty. Specifically, the PCM-Adapter comprises two sequential modules for progressive probabilistic alignment. We first devise a Mahalanobis Alignment Module (MAM) that models textual tokens as Gaussian distributions and computes patch-text compatibility via Mahalanobis distance, yielding variance-conditioned matching that downweights unreliable feature dimensions. Moreover, we devise a Distribution Flow Module (DFM) that estimates modality-wise confidence parameters and performs vision-guided refinement of textual distributions, accommodating distributional variation across imaging modalities. Extensive experiments across eight medical segmentation benchmarks demonstrate that DistMedVL outperforms state-of-the-art methods with only 6.3M trainable parameters, exhibiting superior data efficiency, perturbation robustness and cross-dataset generalization.

Problem

• uncertainty-aware
• medical image segmentation
• vision-language alignment
• cross-modal matching
• domain shift

Innovation

• probabilistic vision-language alignment
• uncertainty-aware segmentation
• Mahalanobis distance
• cross-modal adapter
• medical image segmentation