FedTaste: Topology-Aware Structural Transfer for Multimodal Federated Learning with Missing Modalities

2026-07-25

AI Summary

This work addresses the challenges of representation drift and impaired collaboration in multimodal federated learning caused by arbitrary modality missingness and non-IID data distributions. To this end, the authors propose FedTaste, a novel framework that eschews conventional feature alignment or generative imputation strategies. Instead, FedTaste constructs a globally shared multimodal topological blueprint: clients extract topology using frozen foundation models, the server aggregates these structural blueprints, and lightweight modality-adaptive structural prompts—regularized by spectral consistency—are deployed to enable efficient adaptation at clients with missing modalities. Extensive experiments demonstrate that FedTaste consistently outperforms existing methods across diverse datasets and non-IID settings while significantly reducing communication overhead, thereby achieving a favorable trade-off among privacy preservation, efficiency, and model performance.

Abstract

Multimodal Federated Learning is often challenged by arbitrary modality missingness and Non-IID data distributions, which lead to severe representation drift and hinder effective collaboration across clients. Existing methods typically rely on generative imputation, external auxiliary data, or isolated unimodal training to bridge modality gaps, often incurring substantial communication and computational costs as well as potential privacy risks. To address these limitations, we propose FedTaste, a parameter-efficient framework for topology-aware structural transfer in Multimodal Federated Learning with missing modalities. Instead of aligning fragile first-order features, FedTaste focuses on more stable group-level semantic relations. Specifically, FedTaste leverages frozen foundation models to extract a joint multimodal topology from full-modality clients, which is then consolidated by the server into a global structural blueprint. To adapt clients with missing modalities, we introduce Modality-Adaptive Structural Prompts together with spectral consistency regularization, enabling lightweight branch-specific adaptation that aligns local partial representations with the shared blueprint. In this way, FedTaste avoids explicit modality imputation while preserving shared semantic structure across clients. Extensive experiments demonstrate that FedTaste consistently achieves superior performance across multiple datasets and challenging Non-IID settings, while substantially reducing communication overhead compared with existing methods.

Problem

• Multimodal Federated Learning
• Missing Modalities
• Non-IID Data
• Representation Drift
• Modality Missingness

Innovation

• topology-aware transfer
• structural prompt
• modality missingness
• federated learning
• non-IID