Towards Privacy-Preserving Federated Prompt Tuning under Data Heterogeneity: A Subspace-Decomposed Expert Approach

2026-07-23

AI Summary

This work addresses the challenge of balancing personalization, generalization, privacy preservation, and communication efficiency in federated learning under data heterogeneity. To this end, the authors propose FedSEPT, a novel framework that introduces Subspace-based Expert Modeling (SEM) to decompose prompt parameters into a shared low-rank factor, a common basis, and private residuals. This decomposition substantially reduces communication overhead and narrows the noise space required for differential privacy. Additionally, FedSEPT incorporates an Instance-aware Expert Fusion (IEF) mechanism that enables adaptive composition of semantically complementary experts on each client. Extensive experiments across 11 heterogeneous benchmarks demonstrate that FedSEPT consistently outperforms strong baselines, achieving a superior trade-off between local adaptability and global generalization under identical privacy constraints.

Abstract

Federated prompt tuning (FPT) enables collaborative adaptation of vision--language models (VLMs) using lightweight prompts. Existing methods often address heterogeneity and privacy through a split-prompt design under local differential privacy (DP), combining a shared prompt for global transfer with private prompts for local adaptation. However, a single shared prompt may over-smooth diverse transferable knowledge, weakening the balance between personalization and generalization. Multi-expert prompts (MEPs) can better capture this diversity, but enlarge the communicated space, increasing DP noise and communication cost while making robust expert composition more difficult. We propose FedSEPT, a privacy-preserving Fed}erated Subspace-decomposed Expert Prompt Tuning. Specifically, we employ Subspace-decomposed Expert Modeling (SEM) to parameterize multiple prompt experts with shared low-rank factors, a fixed public basis, and private residuals, thereby confining communication and DP perturbation to a compact factor space while enabling direct server aggregation in a common coordinate system. We further design Instance-aware Expert Fusion (IEF), which adaptively combines semantically complementary experts via on-device routing and performs efficient logit-level fusion using cached expert-specific text features. Extensive experiments on 11 heterogeneous benchmarks show that, under the same privacy constraints, FedSEPT achieves a better trade-off between local adaptation and global generalization than strong baselines.

Problem

• Federated Prompt Tuning
• Data Heterogeneity
• Privacy Preservation
• Multi-Expert Prompts
• Differential Privacy

Innovation

• Federated Prompt Tuning
• Subspace-Decomposed Experts
• Local Differential Privacy
• Low-Rank Parameterization
• Instance-aware Fusion