Multi-Modal Style Transfer-based Prompt Tuning for Efficient Federated Domain Generalization

2026-01-09

arXiv.org

AI Summary

This work proposes FaST-PT, a novel framework for federated domain generalization that addresses domain shift caused by cross-client data heterogeneity and the associated high communication and computational overhead. The approach introduces lightweight multimodal style transfer guided by textual supervision to enhance local feature representations and incorporates a dual-prompt architecture that disentangles global and domain-specific knowledge. Coupled with a sample-adaptive, domain-aware prompt generation mechanism, FaST-PT effectively reduces communication costs while improving generalization to unseen domains. Extensive experiments on four benchmarks—including PACS and DomainNet—demonstrate significant performance gains over state-of-the-art methods such as FedDG-GA and DiPrompt. Ablation studies further confirm the framework’s effectiveness and efficiency.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Federated Domain Generalization (FDG) aims to collaboratively train a global model across distributed clients that can generalize well on unseen domains. However, existing FDG methods typically struggle with cross-client data heterogeneity and incur significant communication and computation overhead. To address these challenges, this paper presents a new FDG framework, dubbed FaST-PT, which facilitates local feature augmentation and efficient unseen domain adaptation in a distributed manner. First, we propose a lightweight Multi-Modal Style Transfer (MST) method to transform image embedding under text supervision, which could expand the training data distribution and mitigate domain shift. We then design a dual-prompt module that decomposes the prompt into global and domain prompts. Specifically, global prompts capture general knowledge from augmented embedding across clients, while domain prompts capture domain-specific knowledge from local data. Besides, Domain-aware Prompt Generation (DPG) is introduced to adaptively generate suitable prompts for each sample, which facilitates unseen domain adaptation through knowledge fusion. Extensive experiments on four cross-domain benchmark datasets, e.g., PACS and DomainNet, demonstrate the superior performance of FaST-PT over SOTA FDG methods such as FedDG-GA and DiPrompt. Ablation studies further validate the effectiveness and efficiency of FaST-PT.

Problem

• Federated Domain Generalization
• data heterogeneity
• unseen domain generalization
• communication overhead
• computation overhead

Innovation

• Multi-Modal Style Transfer
• Prompt Tuning
• Federated Domain Generalization
• Domain-aware Prompt Generation
• Dual-prompt Module