FedSocket: Recipient-Executable Knowledge Exchange for Heterogeneous Multimodal Federated Learning

2026-09-29

AI Summary

This study addresses the challenge in federated learning where cross-client knowledge cannot be directly executed by receivers due to heterogeneity in modalities, architectures, and tasks. To overcome this, we propose a multimodal federated learning framework centered on receiver executability. Specifically, a shared Q-matrix is introduced to bridge heterogeneous private models, enabling cross-client knowledge transfer by exchanging only parameters and counts. Furthermore, an ownership-aware aggregation mechanism and a unified prediction interface are constructed, incorporating local teacher guidance to support joint inference. Experimental evaluations on the MELD and UCF-51 datasets demonstrate that the proposed approach improves accuracy under missing-modality scenarios by 14.44% and 15.51%, respectively, significantly outperforming independent ensemble baselines.

Abstract

Federated knowledge must remain usable by recipients with different modalities, private architectures, and tasks. We present FedSocket, which makes recipient execution a design requirement of the exchanged model. A shared Q combines recipient-computable inputs, task-owned outputs, and ownership-aware aggregation, connecting heterogeneous private models through a common prediction interface. Private models teach local Q copies; the returned Q supports local learning and Joint inference, with only Q parameters and counts exchanged. Across six datasets, FedSocket improves missing-modality recipient accuracy over Local by 14.44 and 15.51 percentage points on MELD and UCF-51. Under matched inference capacity, Joint exceeds independent ensembles by 11.06 points in UCF-51 accuracy and 4.87 points in mean bidirectional Flickr30k R@1. Joint also improves over Q alone on all four heterogeneous endpoints, demonstrating the value of combining local and exchanged predictions. Teacher controls, sharing-path interventions, and component factorials identify the roles of supervision, sharing, and deployment. FedSocket makes exchanged knowledge directly usable from federated training to recipient inference.

Problem

• Federated Learning
• Multimodal
• Heterogeneous Models
• Knowledge Exchange
• Missing Modality

Innovation

• Heterogeneous Federated Learning
• Multimodal Learning
• Knowledge Exchange
• Joint Inference
• Missing Modality