Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptation

2025-08-22

AI Summary

Adapting foundation models (FMs) to edge devices in privacy-sensitive scenarios is challenging due to strict data locality constraints, scarce labeled data, limited on-device compute resources, and misalignment across heterogeneous multi-resolution inputs. Method: We propose PSSFL, a practical semi-supervised federated learning framework. Its core is FedMox—a federated mixture-of-experts architecture featuring a spatial router to align cross-device multi-scale features, a Soft-Mixture strategy to stabilize semi-supervised training, and a sparse expert mechanism to reduce edge memory overhead. Contribution/Results: PSSFL achieves, for the first time, high-performance FM adaptation under extremely low labeling rates (e.g., <1%). End-to-end evaluation on a real-world autonomous driving dataset demonstrates significant improvements in object detection accuracy, a 32% reduction in edge memory footprint, strong privacy preservation (via data-free federation), high scalability, and practical deployability.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Computer Vision: Multi-modal Vision
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

Foundation models (FMs) exhibit remarkable generalization but require adaptation to downstream tasks, particularly in privacy-sensitive applications. Due to data privacy regulations, cloud-based FMs cannot directly access private edge data, limiting their adaptation. Federated learning (FL) provides a privacy-aware alternative, but existing FL approaches overlook the constraints imposed by edge devices -- namely, limited computational resources and the scarcity of labeled data. To address these challenges, we introduce Practical Semi-Supervised Federated Learning (PSSFL), where edge devices hold only unlabeled, low-resolution data, while the server has limited labeled, high-resolution data. In this setting, we propose the Federated Mixture of Experts (FedMox), a novel framework that enhances FM adaptation in FL. FedMox tackles computational and resolution mismatch challenges via a sparse Mixture-of-Experts architecture, employing a spatial router to align features across resolutions and a Soft-Mixture strategy to stabilize semi-supervised learning. We take object detection as a case study, and experiments on real-world autonomous driving datasets demonstrate that FedMox effectively adapts FMs under PSSFL, significantly improving performance with constrained memory costs on edge devices. Our work paves the way for scalable and privacy-preserving FM adaptation in federated scenarios.

Problem

• Adapting foundation models with privacy constraints
• Addressing computational limits on edge devices
• Overcoming scarcity of labeled data in federated learning

Innovation

• Federated Mixture of Experts framework
• Spatial router for resolution alignment
• Soft-Mixture strategy for semi-supervised learning