Improving Group Connectivity for Generalization of Federated Deep Learning

2024-02-29

arXiv.org

AI Summary

In federated learning, heterogeneous clients’ local model aggregation severely degrades global model generalization compared to centralized training. To address this, we introduce FedGuCci(+), the first framework extending Linear Mode Connectivity (LMC) to multi-model group settings, grounded in parameter-space “group connectivity” and guaranteed to preserve transitivity. Methodologically, FedGuCci(+) establishes anchor-based interpolation connectivity analysis, differentiable group connectivity metrics, and connectivity-driven fusion optimization—fully compatible with modern architectures including ViTs and PLMs. Evaluated on four CV and six NLP benchmarks, it achieves consistent average accuracy gains of 1.8–3.5 percentage points across diverse heterogeneity levels. Both theoretical analysis and empirical results jointly validate that explicit connectivity modeling fundamentally enhances generalization.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Large Vision Models
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Federated learning (FL) involves multiple heterogeneous clients collaboratively training a global model via iterative local updates and model fusion. The generalization of FL's global model has a large gap compared with centralized training, which is its bottleneck for broader applications. In this paper, we study and improve FL's generalization through a fundamental"connectivity'' perspective, which means how the local models are connected in the parameter region and fused into a generalized global model. The term"connectivity'' is derived from linear mode connectivity (LMC), studying the interpolated loss landscape of two different solutions (e.g., modes) of neural networks. Bridging the gap between LMC and FL, in this paper, we leverage fixed anchor models to empirically and theoretically study the transitivity property of connectivity from two models (LMC) to a group of models (model fusion in FL). Based on the findings, we propose FedGuCci(+), improving group connectivity for better generalization. It is shown that our methods can boost the generalization of FL under client heterogeneity across various tasks (4 CV datasets and 6 NLP datasets) and model architectures (e.g., ViTs and PLMs).

Problem

• Improving federated learning generalization via connectivity analysis
• Bridging gap between linear mode connectivity and model fusion
• Enhancing group connectivity for heterogeneous client performance

Innovation

• Leverages fixed anchor models for connectivity
• Improves group connectivity for better generalization
• Boosts FL generalization under client heterogeneity