Enhancing Visual Representation with Textual Semantics: Textual Semantics-Powered Prototypes for Heterogeneous Federated Learning

2025-03-16

AI Summary

To address the degradation of global prototype quality in heterogeneous federated learning (FL) caused by semantic relationship loss, this paper introduces textual semantic priors into federated prototype learning for the first time. Specifically, fine-grained class descriptions are generated using large language models and encoded via a pretrained language model to construct semantically enhanced text prototypes. A learnable cross-modal prompting mechanism is further designed to bridge the modality gap between vision and language, enabling adaptive alignment of text prototypes to client-side visual tasks. The proposed method explicitly models and preserves inter-class semantic structure. Evaluated on multiple heterogeneous FL benchmarks, it achieves average accuracy improvements of 3.2–5.8%, accelerates convergence by over 40%, and demonstrates superior generalization performance.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Language and Vision

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Federated Prototype Learning (FedPL) has emerged as an effective strategy for handling data heterogeneity in Federated Learning (FL). In FedPL, clients collaboratively construct a set of global feature centers (prototypes), and let local features align with these prototypes to mitigate the effects of data heterogeneity. The performance of FedPL highly depends on the quality of prototypes. Existing methods assume that larger inter-class distances among prototypes yield better performance, and thus design different methods to increase these distances. However, we observe that while these methods increase prototype distances to enhance class discrimination, they inevitably disrupt essential semantic relationships among classes, which are crucial for model generalization. This raises an important question: how to construct prototypes that inherently preserve semantic relationships among classes? Directly learning these relationships from limited and heterogeneous client data can be problematic in FL. Recently, the success of pre-trained language models (PLMs) demonstrates their ability to capture semantic relationships from vast textual corpora. Motivated by this, we propose FedTSP, a novel method that leverages PLMs to construct semantically enriched prototypes from the textual modality, enabling more effective collaboration in heterogeneous data settings. We first use a large language model (LLM) to generate fine-grained textual descriptions for each class, which are then processed by a PLM on the server to form textual prototypes. To address the modality gap between client image models and the PLM, we introduce trainable prompts, allowing prototypes to adapt better to client tasks. Extensive experiments demonstrate that FedTSP mitigates data heterogeneity while significantly accelerating convergence.

Problem

• Improving prototype quality in Federated Prototype Learning (FedPL).
• Preserving semantic relationships among classes in heterogeneous data.
• Leveraging pre-trained language models to enhance prototype construction.

Innovation

• Leverages pre-trained language models for semantic enrichment
• Uses textual prototypes to preserve class relationships
• Introduces trainable prompts to bridge modality gaps