FLEET: A Federated Learning Emulation and Evaluation Testbed for Holistic Research

2025-08-30

AI Summary

Existing federated learning (FL) evaluation platforms fail to accurately reflect algorithmic performance under dynamic network conditions, primarily due to oversimplified modeling of client heterogeneity and the tight coupling between FL execution and time-varying network states—such as bandwidth fluctuations, latency spikes, packet loss, and background traffic. To address this, we propose the first integrated testbed that jointly models high-fidelity network dynamics and FL algorithm execution. It supports multi-framework integration, programmable topologies, end-to-end experiment configuration, and cross-scenario reproducible evaluation. Our key innovation lies in a framework-agnostic learning component interface that tightly couples dynamic network simulation—including real-time traffic generation—with the FL training pipeline, enabling joint quantification of convergence behavior and communication efficiency. Experimental results demonstrate that our platform significantly enhances the realism, systematicity, and generalizability of FL evaluation.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Algorithm Configuration
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Federated Learning (FL) presents a robust paradigm for privacy-preserving, decentralized machine learning. However, a significant gap persists between the theoretical design of FL algorithms and their practical performance, largely because existing evaluation tools often fail to model realistic operational conditions. Many testbeds oversimplify the critical dynamics among algorithmic efficiency, client-level heterogeneity, and continuously evolving network infrastructure. To address this challenge, we introduce the Federated Learning Emulation and Evaluation Testbed (FLEET). This comprehensive platform provides a scalable and configurable environment by integrating a versatile, framework-agnostic learning component with a high-fidelity network emulator. FLEET supports diverse machine learning frameworks, customizable real-world network topologies, and dynamic background traffic generation. The testbed collects holistic metrics that correlate algorithmic outcomes with detailed network statistics. By unifying the entire experiment configuration, FLEET enables researchers to systematically investigate how network constraints, such as limited bandwidth, high latency, and packet loss, affect the convergence and efficiency of FL algorithms. This work provides the research community with a robust tool to bridge the gap between algorithmic theory and real-world network conditions, promoting the holistic and reproducible evaluation of federated learning systems.

Problem

• Bridging gap between FL theory and practical performance
• Modeling realistic network conditions in FL evaluation
• Investigating network constraints on FL convergence efficiency

Innovation

• Federated Learning Emulation and Evaluation Testbed
• Integrates framework-agnostic learning with network emulator
• Measures algorithmic outcomes with detailed network statistics