Distributed optimization: designed for federated learning

2025-08-11

AI Summary

To address the challenges of high statistical heterogeneity, diverse communication topologies, and stringent privacy constraints in cross-organizational federated learning (FL), this paper proposes a generic distributed optimization algorithm grounded in the augmented Lagrangian framework. Methodologically, it integrates proximal relaxation with quadratic approximation techniques, enabling unified convergence analysis for variants including proximal gradient descent and stochastic gradient descent—supporting both centralized and decentralized topologies, asynchronous communication, and non-IID data. Theoretically, it establishes the first general convergence analysis framework compatible with multiple FL architectures and termination criteria. Empirically, the algorithm achieves significantly faster convergence and improved communication efficiency in large-scale, highly heterogeneous settings, while demonstrating robustness and practical applicability.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Non-convex Optimization
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

Federated Learning (FL), as a distributed collaborative Machine Learning (ML) framework under privacy-preserving constraints, has garnered increasing research attention in cross-organizational data collaboration scenarios. This paper proposes a class of distributed optimization algorithms based on the augmented Lagrangian technique, designed to accommodate diverse communication topologies in both centralized and decentralized FL settings. Furthermore, we develop multiple termination criteria and parameter update mechanisms to enhance computational efficiency, accompanied by rigorous theoretical guarantees of convergence. By generalizing the augmented Lagrangian relaxation through the incorporation of proximal relaxation and quadratic approximation, our framework systematically recovers a broad of classical unconstrained optimization methods, including proximal algorithm, classic gradient descent, and stochastic gradient descent, among others. Notably, the convergence properties of these methods can be naturally derived within the proposed theoretical framework. Numerical experiments demonstrate that the proposed algorithm exhibits strong performance in large-scale settings with significant statistical heterogeneity across clients.

Problem

• Develop distributed optimization for federated learning privacy constraints
• Propose algorithms for diverse FL communication topologies
• Enhance efficiency with termination criteria and convergence guarantees

Innovation

• Augmented Lagrangian technique for distributed optimization
• Proximal relaxation and quadratic approximation generalization
• Multiple termination criteria for efficiency enhancement