Federated Loss Exploration for Improved Convergence on Non-IID Data

2025-06-23

AI Summary

To address slow convergence and poor robustness in federated learning (FL) under non-IID data, this paper proposes FedLEx. The method dynamically constructs a lightweight global guidance matrix based on client-wise gradient deviation—without requiring data sharing or prior distribution assumptions—enabling adaptive optimization guidance over complex loss landscapes. FedLEx integrates gradient deviation modeling, federated aggregation, and few-shot/few-round local training to establish an efficient knowledge transfer mechanism. Extensive experiments demonstrate that FedLEx significantly accelerates convergence and improves final model accuracy across multiple mainstream FL algorithms, especially under realistic non-IID settings. It achieves superior computational efficiency and strong generalization robustness, outperforming existing approaches in both heterogeneous data scenarios and resource-constrained environments.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing

Abstract

Federated learning (FL) has emerged as a groundbreaking paradigm in machine learning (ML), offering privacy-preserving collaborative model training across diverse datasets. Despite its promise, FL faces significant hurdles in non-identically and independently distributed (non-IID) data scenarios, where most existing methods often struggle with data heterogeneity and lack robustness in performance. This paper introduces Federated Loss Exploration (FedLEx), an innovative approach specifically designed to tackle these challenges. FedLEx distinctively addresses the shortcomings of existing FL methods in non-IID settings by optimizing its learning behavior for scenarios in which assumptions about data heterogeneity are impractical or unknown. It employs a federated loss exploration technique, where clients contribute to a global guidance matrix by calculating gradient deviations for model parameters. This matrix serves as a strategic compass to guide clients' gradient updates in subsequent FL rounds, thereby fostering optimal parameter updates for the global model. FedLEx effectively navigates the complex loss surfaces inherent in non-IID data, enhancing knowledge transfer in an efficient manner, since only a small number of epochs and small amount of data are required to build a strong global guidance matrix that can achieve model convergence without the need for additional data sharing or data distribution statics in a large client scenario. Our extensive experiments with state-of-the art FL algorithms demonstrate significant improvements in performance, particularly under realistic non-IID conditions, thus highlighting FedLEx's potential to overcome critical barriers in diverse FL applications.

Problem

• Addresses performance issues in federated learning with non-IID data.
• Optimizes learning behavior without data heterogeneity assumptions.
• Enhances model convergence using a global guidance matrix.

Innovation

• Federated Loss Exploration (FedLEx) technique
• Global guidance matrix for gradient updates
• Efficient convergence on non-IID data