FLASH: Federated Learning Across Simultaneous Heterogeneities

2024-02-13

arXiv.org

AI Summary

Federated learning faces concurrent heterogeneity across three dimensions: statistical (non-IID data distributions), data-quality (noisy or low-fidelity local datasets), and system (device-specific computation and communication delays), which existing methods struggle to jointly model and optimize. To address this, we propose a lightweight, flexible context-aware client selection framework. First, we unify the three heterogeneity types into a dynamic contextual representation and formulate client selection as a Contextual Multi-Armed Bandit (CMAB) problem, enabling heterogeneity-aware scoring and selection. Second, we introduce a lightweight coordination protocol to minimize system overhead. Our framework explicitly balances statistical utility and system efficiency. Extensive experiments demonstrate up to 10 percentage points absolute accuracy improvement over state-of-the-art methods across diverse heterogeneous settings. Moreover, it is fully compatible with and enhances mainstream robust aggregation algorithms—accelerating convergence and improving final model performance.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Learning to Search
• Multiagent Systems: Multiagent Learning

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing

Abstract

The key premise of federated learning (FL) is to train ML models across a diverse set of data-owners (clients), without exchanging local data. An overarching challenge to this date is client heterogeneity, which may arise not only from variations in data distribution, but also in data quality, as well as compute/communication latency. An integrated view of these diverse and concurrent sources of heterogeneity is critical; for instance, low-latency clients may have poor data quality, and vice versa. In this work, we propose FLASH(Federated Learning Across Simultaneous Heterogeneities), a lightweight and flexible client selection algorithm that outperforms state-of-the-art FL frameworks under extensive sources of heterogeneity, by trading-off the statistical information associated with the client's data quality, data distribution, and latency. FLASH is the first method, to our knowledge, for handling all these heterogeneities in a unified manner. To do so, FLASH models the learning dynamics through contextual multi-armed bandits (CMAB) and dynamically selects the most promising clients. Through extensive experiments, we demonstrate that FLASH achieves substantial and consistent improvements over state-of-the-art baselines -- as much as 10% in absolute accuracy -- thanks to its unified approach. Importantly, FLASH also outperforms federated aggregation methods that are designed to handle highly heterogeneous settings and even enjoys a performance boost when integrated with them.

Problem

• Handling simultaneous client heterogeneities in federated learning
• Addressing data quality, distribution, and latency trade-offs
• Unifying diverse heterogeneity sources through dynamic client selection

Innovation

• Lightweight client selection algorithm
• Unified handling of multiple heterogeneities
• Contextual multi-armed bandits modeling