FedWSQ: Efficient Federated Learning with Weight Standardization and Distribution-Aware Non-Uniform Quantization

2025-06-30

AI Summary

Federated learning faces two critical challenges: model bias induced by statistical heterogeneity (non-IID data) and excessive communication overhead. To address these, we propose FedWSQ—a novel framework integrating Weight Standardization (WS) and Distribution-Aware Non-Uniform Quantization (DANUQ). WS mitigates bias by normalizing local model updates across clients, while DANUQ dynamically constructs non-uniform quantization intervals based on local gradient/weight distributions, preserving information fidelity even at ultra-low bitwidths (e.g., 1–2 bits). Evaluated on CIFAR-10/100 and Tiny-ImageNet under extreme non-IID settings and stringent bandwidth constraints, FedWSQ achieves 3.2–7.8% higher accuracy than state-of-the-art baselines and reduces communication volume to only 12–18% of the original. To the best of our knowledge, this is the first approach to jointly optimize for both high robustness against data heterogeneity and ultra-efficient transmission.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Federated learning (FL) often suffers from performance degradation due to key challenges such as data heterogeneity and communication constraints. To address these limitations, we present a novel FL framework called FedWSQ, which integrates weight standardization (WS) and the proposed distribution-aware non-uniform quantization (DANUQ). WS enhances FL performance by filtering out biased components in local updates during training, thereby improving the robustness of the model against data heterogeneity and unstable client participation. In addition, DANUQ minimizes quantization errors by leveraging the statistical properties of local model updates. As a result, FedWSQ significantly reduces communication overhead while maintaining superior model accuracy. Extensive experiments on FL benchmark datasets demonstrate that FedWSQ consistently outperforms existing FL methods across various challenging FL settings, including extreme data heterogeneity and ultra-low-bit communication scenarios.

Problem

• Addresses performance degradation in federated learning from data heterogeneity
• Reduces communication overhead with efficient weight standardization and quantization
• Improves model accuracy in extreme data heterogeneity and low-bit scenarios

Innovation

• Weight standardization filters biased local updates
• Distribution-aware non-uniform quantization minimizes errors
• Reduces communication overhead while maintaining accuracy