Prototype-guided Bilateral Alignment Multimodal Federated Learning

2026-09-30

AI Summary

This study addresses the performance bottlenecks in multimodal federated learning caused by model heterogeneity and modality imbalance. To this end, we propose MFedPBA, a novel framework that introduces a pioneering bi-level alignment mechanism. At the feature level, it achieves projection alignment across heterogeneous feature spaces by integrating contrastive learning with the Gromov-Wasserstein distance. At the decision level, it designs an entropy-weighted logit prototype aggregation strategy to facilitate cross-client knowledge fusion. Experimental results demonstrate that the proposed method significantly outperforms existing state-of-the-art baselines under scenarios involving model heterogeneity and modality imbalance. By effectively mitigating these challenges, MFedPBA establishes a robust and efficient collaborative paradigm for multimodal federated learning.

Abstract

Multimodal federated learning (MFL) has emerged as a pivotal paradigm for leveraging distributed data to enhance model performance. However, existing methods predominantly rely on idealized assumptions of model homogeneity and balanced modality distributions, rendering them ill-suited for practical scenarios characterized by heterogeneous client architectures and severe modality imbalance. To address these challenges, we propose a \textbf{M}ultimodal \textbf{Fed}erated learning Prototype-guided Bilateral Alignment (MFedPBA) framework. MFedPBA facilitates robust knowledge synergy through a dual alignment mechanism: (i) at the feature level, it aligns heterogeneous feature spaces via a projection encoder optimized by contrastive learning and the Gromov-Wasserstein distance; (ii) at the decision level, it employs an entropy-weighted aggregation of naturally aligned logit prototypes. This novel design achieves robust MFL by jointly tackling heterogeneous feature spaces and collectively aggregating decisions. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art baselines under conditions of model heterogeneity and modality imbalance.

Problem

• Multimodal Federated Learning
• Model Heterogeneity
• Modality Imbalance
• Feature Alignment

Innovation

• Multimodal Federated Learning
• Prototype-guided Bilateral Alignment
• Gromov-Wasserstein Distance
• Model Heterogeneity
• Modality Imbalance