pFedLVM: A Large Vision Model (LVM)-Driven and Latent Feature-Based Personalized Federated Learning Framework in Autonomous Driving

2024-05-07

arXiv.org

AI Summary

To address poor model generalization in autonomous driving caused by data heterogeneity and domain shift, as well as key bottlenecks of conventional federated learning—including underfitting on incremental data, prohibitive on-device deployment costs for large vision models (LVMs), high communication overhead, and neglect of client-specific characteristics—this paper proposes pFedLVM: a personalized federated learning framework wherein the LVM resides exclusively on the server, while clients upload only lightweight latent features. Its core innovations include the first “server-side LVM + client-side lightweight encoder” paradigm, a shared-personal dual-path feature fusion mechanism, and a co-optimized loss function. Evaluated across multiple autonomous driving perception tasks, pFedLVM significantly outperforms state-of-the-art federated methods, reducing communication overhead by 83%, decreasing per-vehicle inference latency by 91%, and improving personalized mean average precision (mAP) by 6.2%.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Deep learning-based Autonomous Driving (AD) models often exhibit poor generalization due to data heterogeneity in an ever domain-shifting environment. While Federated Learning (FL) could improve the generalization of an AD model (known as FedAD system), conventional models often struggle with under-fitting as the amount of accumulated training data progressively increases. To address this issue, instead of conventional small models, employing Large Vision Models (LVMs) in FedAD is a viable option for better learning of representations from a vast volume of data. However, implementing LVMs in FedAD introduces three challenges: (I) the extremely high communication overheads associated with transmitting LVMs between participating vehicles and a central server; (II) lack of computing resource to deploy LVMs on each vehicle; (III) the performance drop due to LVM focusing on shared features but overlooking local vehicle characteristics. To overcome these challenges, we propose pFedLVM, a LVM-Driven, Latent Feature-Based Personalized Federated Learning framework. In this approach, the LVM is deployed only on central server, which effectively alleviates the computational burden on individual vehicles. Furthermore, the exchange between central server and vehicles are the learned features rather than the LVM parameters, which significantly reduces communication overhead. In addition, we utilize both shared features from all participating vehicles and individual characteristics from each vehicle to establish a personalized learning mechanism. This enables each vehicle's model to learn features from others while preserving its personalized characteristics, thereby outperforming globally shared models trained in general FL. Extensive experiments demonstrate that pFedLVM outperforms the existing state-of-the-art approaches.

Problem

• Addresses poor generalization in autonomous driving models due to data heterogeneity.
• Reduces communication overhead and computational burden in federated learning with large vision models.
• Enhances personalized learning by integrating shared and local vehicle characteristics.

Innovation

• Uses Large Vision Models on central server
• Exchanges learned features, not model parameters
• Combines shared and individual vehicle features