Towards Instance-wise Personalized Federated Learning via Semi-Implicit Bayesian Prompt Tuning

2025-08-27

AI Summary

Existing personalized federated learning (pFL) methods assume client data follow a single distribution and learn only client-level models, rendering them inadequate for real-world scenarios where intra-client multi-source or multi-domain data induce instance-level heterogeneity. To address this, we propose a fine-grained, instance-level adaptive modeling framework—the first to integrate Bayesian prompt tuning into pFL. Specifically, we model the prompt posterior distribution via implicit variational inference, enabling sample-dependent personalized prompt generation. By unifying visual prompt tuning with semi-implicit variational inference, our approach enhances generalization under both feature and label heterogeneity. Extensive experiments on multiple benchmark datasets demonstrate significant improvements over state-of-the-art pFL methods, validating the robustness and effectiveness of our framework in handling complex, fine-grained data heterogeneity.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Diffusion Models for Vision

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Federated learning (FL) is a privacy-preserving machine learning paradigm that enables collaborative model training across multiple distributed clients without disclosing their raw data. Personalized federated learning (pFL) has gained increasing attention for its ability to address data heterogeneity. However, most existing pFL methods assume that each client's data follows a single distribution and learn one client-level personalized model for each client. This assumption often fails in practice, where a single client may possess data from multiple sources or domains, resulting in significant intra-client heterogeneity and suboptimal performance. To tackle this challenge, we propose pFedBayesPT, a fine-grained instance-wise pFL framework based on visual prompt tuning. Specifically, we formulate instance-wise prompt generation from a Bayesian perspective and model the prompt posterior as an implicit distribution to capture diverse visual semantics. We derive a variational training objective under the semi-implicit variational inference framework. Extensive experiments on benchmark datasets demonstrate that pFedBayesPT consistently outperforms existing pFL methods under both feature and label heterogeneity settings.

Problem

• Addresses intra-client data heterogeneity in federated learning
• Develops instance-wise personalized models via Bayesian prompt tuning
• Improves performance under feature and label heterogeneity settings

Innovation

• Semi-implicit Bayesian prompt tuning
• Instance-wise personalized federated learning
• Variational training objective framework