Analytic Federated Learning

2024-05-25

arXiv.org

AI Summary

To address the bottlenecks of large communication overhead, slow convergence, and hyperparameter sensitivity in federated learning (FL) with pretrained models under highly non-IID data and massive client scales (≥1,000), this paper proposes Analytic Federated Learning (AFL). AFL is the first FL framework to yield a weight-invariant closed-form solution: clients perform only a single round of local forward computation, and the server conducts one round of Absolute Aggregation (AA), completely eliminating gradient descent, backpropagation, and iterative optimization. Its core contributions include theoretically guaranteed absolute convergence, generalization robustness independent of data heterogeneity and client count, and truly hyperparameter-free operation. Experiments demonstrate that AFL significantly outperforms state-of-the-art methods under extreme non-IID settings and thousand-scale client deployments—achieving faster convergence, higher accuracy, and substantially reduced communication and computational costs.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Multiagent Systems: Multiagent Learning
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

In this paper, we introduce analytic federated learning (AFL), a new training paradigm that brings analytical (i.e., closed-form) solutions to the federated learning (FL) community. Our AFL draws inspiration from analytic learning -- a gradient-free technique that trains neural networks with analytical solutions in one epoch. In the local client training stage, the AFL facilitates a one-epoch training, eliminating the necessity for multi-epoch updates. In the aggregation stage, we derive an absolute aggregation (AA) law. This AA law allows a single-round aggregation, removing the need for multiple aggregation rounds. More importantly, the AFL exhibits a extit{weight-invariant} property, meaning that regardless of how the full dataset is distributed among clients, the aggregated result remains identical. This could spawn various potentials, such as data heterogeneity invariance, client-number invariance, absolute convergence, and being hyperparameter-free (our AFL is the first hyperparameter-free method in FL history). We conduct experiments across various FL settings including extremely non-IID ones, and scenarios with a large number of clients (e.g., $ge 1000$). In all these settings, our AFL constantly performs competitively while existing FL techniques encounter various obstacles. Code is available at url{https://github.com/ZHUANGHP/Analytic-federated-learning}

Problem

• Proposes AFL for federated learning with pre-trained models
• Enables single-round aggregation to reduce communication overhead
• Achieves invariance to data partitioning and heterogeneity

Innovation

• Single-round closed-form federated learning solution
• One-epoch local training for efficiency
• Absolute aggregation law reduces communication overhead