On Provable Benefits of Muon in Federated Learning

2025-10-04

AI Summary

The performance of the Muon optimizer in federated learning (FL) remains unexplored. Method: This paper introduces Muon to FL for the first time, proposing FedMuon—a novel distributed optimization algorithm. FedMuon orthogonalizes gradient update directions to achieve fully adaptive learning rates without requiring problem-specific parameters (e.g., Lipschitz constants or gradient variance bounds). We establish its convergence for non-convex objectives and prove its robustness against heavy-tailed stochastic noise. Results: Experiments across diverse neural architectures (CNNs, RNNs, ViTs) and heterogeneous data settings demonstrate that FedMuon significantly improves convergence speed and training stability over baselines including FedAvg and FedAdam. Key contributions include: (i) the first federated adaptation of Muon; (ii) a parameter-free adaptive mechanism grounded in orthogonalized updates; and (iii) the first theoretical convergence guarantees for an adaptive FL optimizer under heavy-tailed noise.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The recently introduced optimizer, Muon, has gained increasing attention due to its superior performance across a wide range of applications. However, its effectiveness in federated learning remains unexplored. To address this gap, this paper investigates the performance of Muon in the federated learning setting. Specifically, we propose a new algorithm, FedMuon, and establish its convergence rate for nonconvex problems. Our theoretical analysis reveals multiple favorable properties of FedMuon. In particular, due to its orthonormalized update direction, the learning rate of FedMuon is independent of problem-specific parameters, and, importantly, it can naturally accommodate heavy-tailed noise. The extensive experiments on a variety of neural network architectures validate the effectiveness of the proposed algorithm.

Problem

• Explores Muon optimizer's effectiveness in federated learning
• Proposes FedMuon algorithm with convergence guarantees for nonconvex problems
• Validates algorithm performance across neural architectures under heavy-tailed noise

Innovation

• FedMuon algorithm for federated learning
• Orthonormalized update direction for parameter independence
• Accommodates heavy-tailed noise naturally