Federated Multimodal Learning with Dual Adapters and Selective Pruning for Communication and Computational Efficiency

2025-03-10

AI Summary

To address poor global model generalization and insufficient personalization caused by data heterogeneity in federated learning, this paper proposes a multimodal-aware federated personalization method. We introduce a novel dual-adapter architecture: a large local adapter preserves client-specific modeling capability, while a compact global adapter facilitates cross-device knowledge sharing; further, a parameter-importance-driven selective pruning mechanism dynamically retains critical shared parameters. Evaluated on vision-language joint tasks, our method significantly improves average test accuracy, reduces client-level performance variance and worst-case error, and cuts communication and computational overhead by 30%–50%. The approach thus achieves a balanced trade-off among model generalization, personalization, and deployment efficiency.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

Federated Learning (FL) enables collaborative learning across distributed clients while preserving data privacy. However, FL faces significant challenges when dealing with heterogeneous data distributions, which can lead to suboptimal global models that fail to generalize across diverse clients. In this work, we propose a novel framework designed to tackle these challenges by introducing a dual-adapter approach. The method utilizes a larger local adapter for client-specific personalization and a smaller global adapter to facilitate efficient knowledge sharing across clients. Additionally, we incorporate a pruning mechanism to reduce communication overhead by selectively removing less impactful parameters from the local adapter. Through extensive experiments on a range of vision and language tasks, our method demonstrates superior performance compared to existing approaches. It achieves higher test accuracy, lower performance variance among clients, and improved worst-case performance, all while significantly reducing communication and computation costs. Overall, the proposed method addresses the critical trade-off between model personalization and generalization, offering a scalable solution for real-world FL applications.

Problem

• Addresses challenges in Federated Learning with heterogeneous data distributions.
• Proposes dual-adapter approach for client-specific personalization and efficient knowledge sharing.
• Incorporates selective pruning to reduce communication and computational overhead.

Innovation

• Dual-adapter approach for personalized and shared learning
• Selective pruning to reduce communication overhead
• Improved accuracy and efficiency in federated learning