Coupling Perception and Reasoning in Federated Multimodal Graph Foundation Models

2026-09-24

AI Summary

This study addresses the misalignment between perception and graph reasoning in federated learning caused by frozen multimodal encoders, proposing the FedCORE framework. This method reveals inter-module pairing sensitivity and pioneers the joint optimization of encoders and graph neural networks (GNNs) within a shared low-dimensional latent space. Through federated evolution, FedCORE achieves synergistic adaptation of perception and reasoning while preserving compatibility. Experimental results demonstrate that FedCORE reduces the encoder-GNN pairing gap from 30.6 to 5.9, an 80.7% decrease compared to independent adaptation, thereby effectively enhancing the federated learning performance of multimodal graph neural networks.

Abstract

Federated multimodal graph foundation models (GFMs) aim to adapt pretrained multimodal models to decentralized graph data, where each client owns a private multimodal graph and cannot share raw information. These models typically combine a multimodal Encoder that extracts semantic evidence from heterogeneous modalities and a graph neural network (GNN) that performs relational reasoning over graph structures. However, existing federated GFM adaptation methods mainly update graph-side modules while keeping the multimodal Encoder frozen, limiting adaptation to \emph{how information is propagated} while fixing \emph{what information is extracted}. Through empirical studies, we reveal that Encoder and GNN adaptations are not independent: Encoder adaptation is affected by graph relations, while cross-client module swapping reveals substantial pairing sensitivity between separately parameterized Encoder and GNN updates. Motivated by this observation, we propose \textbf{FedCORE}, a federated adaptation framework that represents Encoder and GNN updates through a shared low-dimensional latent state. FedCORE jointly optimizes this core from multimodal and structural signals and performs federated evolution directly in the shared state space, preserving compatibility between perception and reasoning adaptations. Extensive experiments demonstrate that FedCORE reduces the Encoder--GNN pairing gap from $30.6$ to $5.9$, corresponding to an $80.7\%$ reduction over independent joint adaptation.

Problem

• Federated Learning
• Multimodal Graph Foundation Models
• Perception-Reasoning Coupling
• Encoder-GNN Compatibility

Innovation

• Federated Learning
• Multimodal Graph Foundation Models
• Coupled Adaptation
• Shared Latent State
• Perception-Reasoning Alignment