The Other Side of the Coin: Unveiling the Downsides of Model Aggregation in Federated Learning from a Layer-peeled Perspective

2025-02-05

AI Summary

In federated learning, clients often suffer a sharp drop in local performance after model aggregation, significantly impeding convergence—yet the underlying causes have remained systematically unexplored. This paper introduces the first layer-peeled analysis framework to uncover a dual mechanism: (i) aggregation disrupts feature variability suppression, and (ii) it weakens the coupling between feature representations and subsequent layer parameters. Guided by this insight, we propose a lightweight client-side adaptive recalibration strategy that dynamically restores local adaptability without incurring additional communication overhead. Extensive experiments across diverse datasets and model architectures demonstrate that our method substantially mitigates post-aggregation performance degradation and accelerates convergence. The results empirically validate both the identified causal mechanisms and the effectiveness of our intervention strategy.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Multiagent Systems: Adversarial Agents
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

In federated learning (FL), model aggregation is a critical step by which multiple clients share their knowledge with one another. However, it is also widely recognized that the aggregated model, when sent back to each client, performs poorly on local data until after several rounds of local training. This temporary performance drop can potentially slow down the convergence of the FL model. Most research in FL regards this performance drop as an inherent cost of knowledge sharing among clients and does not give it special attention. While some studies directly focus on designing techniques to alleviate the issue, an in-depth investigation of the reasons behind this performance drop has yet to be conducted.To address this gap, we conduct a layer-peeled analysis of model aggregation across various datasets and model architectures. Our findings reveal that the performance drop can be attributed to two major consequences of the aggregation process: (1) it disrupts feature variability suppression in deep neural networks (DNNs), and (2) it weakens the coupling between features and subsequent parameters.Based on these findings, we propose several simple yet effective strategies to mitigate the negative impacts of model aggregation while still enjoying the benefit it brings. To the best of our knowledge, our work is the first to conduct a layer-peeled analysis of model aggregation, potentially paving the way for the development of more effective FL algorithms.

Problem

• Analyzes performance drop in federated learning
• Investigates model aggregation layer-peeled effects
• Proposes strategies to mitigate aggregation negative impacts

Innovation

• Layer-peeled analysis technique
• Mitigates model aggregation downsides
• Enhances federated learning convergence