MMiC: Mitigating Modality Incompleteness in Clustered Federated Learning

2025-05-11

AI Summary

To address modality missing issues in multimodal federated learning (MFL) caused by heterogeneous data quality or privacy constraints across clients, this paper proposes a cluster-based collaborative mitigation framework that enables efficient and privacy-preserving training. Methodologically, it introduces three key innovations: (1) a novel client selection mechanism integrating the Banzhaf power index to quantitatively assess each client’s contribution to multimodal joint modeling; (2) a dynamic global aggregation strategy inspired by Markowitz portfolio theory, which adaptively compensates for missing modalities at the parameter level; and (3) intra-cluster parameter substitution and collaborative training to enhance model robustness. Extensive experiments on multimodal missing benchmark datasets demonstrate that the proposed approach significantly outperforms existing federated learning baselines, achieving consistent improvements in both global model accuracy and personalized performance while preserving data privacy.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

In the era of big data, data mining has become indispensable for uncovering hidden patterns and insights from vast and complex datasets. The integration of multimodal data sources further enhances its potential. Multimodal Federated Learning (MFL) is a distributed approach that enhances the efficiency and quality of multimodal learning, ensuring collaborative work and privacy protection. However, missing modalities pose a significant challenge in MFL, often due to data quality issues or privacy policies across the clients. In this work, we present MMiC, a framework for Mitigating Modality incompleteness in MFL within the Clusters. MMiC replaces partial parameters within client models inside clusters to mitigate the impact of missing modalities. Furthermore, it leverages the Banzhaf Power Index to optimize client selection under these conditions. Finally, MMiC employs an innovative approach to dynamically control global aggregation by utilizing Markovitz Portfolio Optimization. Extensive experiments demonstrate that MMiC consistently outperforms existing federated learning architectures in both global and personalized performance on multimodal datasets with missing modalities, confirming the effectiveness of our proposed solution.

Problem

• Addressing missing modalities in Multimodal Federated Learning (MFL)
• Optimizing client selection using Banzhaf Power Index
• Dynamically controlling global aggregation with Markovitz Portfolio Optimization

Innovation

• Replaces partial parameters to mitigate missing modalities
• Uses Banzhaf Power Index for optimal client selection
• Dynamically controls aggregation via Markovitz Portfolio Optimization