Where is the Testbed for My Federated Learning Research?

2024-07-19

IFIP International Information Security Conference

AI Summary

Federated learning (FL) lacks systematic, multi-dimensional evaluation under realistic deployment conditions, hindering algorithm selection and practical adoption. Method: We propose CoLExT—the first scalable, low-intrusion testbed for FL research, enabling rapid deployment of custom FL algorithms on heterogeneous edge devices (from Raspberry Pi to smartphones) and real-time collection of accuracy, energy consumption, convergence time, and other key metrics. CoLExT integrates automated instrumentation, cross-platform lightweight agents, distributed experiment orchestration, and real-time visualization. Algorithm integration requires minimal code changes, with instrumentation overhead negligible. Contribution/Results: For the first time, CoLExT reveals latent performance trade-offs, inefficiencies, and implementation flaws of mainstream FL algorithms on real-device clusters. It establishes a reliable empirical benchmark and actionable insights for FL system design, optimization, and deployment.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Security and Privacy: Large-scale security measurements

Abstract

Progressing beyond centralized AI is of paramount importance, yet, distributed AI solutions, in particular various federated learning (FL) algorithms, are often not comprehensively assessed, which prevents the research community from identifying the most promising approaches and practitioners from being convinced that a certain solution is deployment-ready. The largest hurdle towards FL algorithm evaluation is the difficulty of conducting real-world experiments over a variety of FL client devices and different platforms, with different datasets and data distribution, all while assessing various dimensions of algorithm performance, such as inference accuracy, energy consumption, and time to convergence, to name a few. In this paper, we present CoLExT, a real-world testbed for FL research. CoLExT is designed to streamline experimentation with custom FL algorithms in a rich testbed configuration space, with a large number of heterogeneous edge devices, ranging from single-board computers to smartphones, and provides real-time collection and visualization of a variety of metrics through automatic instrumentation. According to our evaluation, porting FL algorithms to CoLExT requires minimal involvement from the developer, and the instrumentation introduces minimal resource usage overhead. Furthermore, through an initial investigation involving popular FL algorithms running on CoLExT, we reveal previously unknown trade-offs, inefficiencies, and programming bugs.

Problem

• Lack of comprehensive evaluation for federated learning algorithms.
• Difficulty in conducting real-world experiments across diverse devices and platforms.
• Need for a testbed to assess FL algorithm performance metrics effectively.

Innovation

• CoLExT testbed for federated learning research
• Supports heterogeneous edge devices and datasets
• Real-time metrics collection with minimal overhead