SMGFM: Spectral Multimodal Graph Pretraining for Multimodal-Attributed Graphs

2026-06-10

AI Summary

This work addresses the challenge in multimodal attributed graphs where structural-induced semantics and modality-intrinsic semantics contribute differently to downstream tasks, yet conventional coupled approaches fail to disentangle them, limiting cross-modal fusion effectiveness. To this end, we propose the first pretraining framework based on graph spectral decomposition: leveraging scalable Chebyshev filters to decompose node signals of each modality into frequency bands, thereby constructing band-resolved modality tokens. We incorporate graph spectral priors to design a frequency-routing mechanism that promotes structural consistency while preserving modality specificity. Furthermore, a topology-conditioned routing strategy evaluates coupling reliability, enabling differentiated interactions across frequency bands and modalities. Extensive experiments demonstrate that our method achieves state-of-the-art performance on multiple multimodal attributed graph benchmarks, significantly improving both graph-level and modality-level task outcomes.

Abstract

Multimodal-attributed graphs (MAGs) couple graph topology with node semantics from text, images, and other modalities. Traditional graph learning contextualizes node semantics by coupling topology with node features. However, this coupling design becomes troublesome in MAGs, where structure-induced and modality-intrinsic semantics may contribute differently to downstream tasks. Structure-induced semantics promote relational consistency through smooth topological variation, whereas modality-intrinsic semantics often encode local, fine-grained distinctions that should not be uniformly smoothed or aligned. Therefore, the key challenge is to identify semantic roles before cross-modal fusion. To this end, we leverage graph-frequency variation as a prior, where low-frequency components capture topology-consistent semantics and high-frequency components preserve modality-specific semantics. Based on this intuition, we propose SMGFM, a spectral multimodal graph pretraining framework that decomposes each modality-specific node signal into graph-frequency bands and assigns band-level semantic roles before cross-modal interaction. Concretely, SMGFM constructs frequency-resolved modality tokens with scalable Chebyshev filters, estimates their coupling reliability through topology-conditioned routing, and performs band-modality interaction before fusion. Its frequency-routed objectives align smooth consensus routes while preserving modality-specific routes, mitigating spatial-domain entanglement and uniform cross-modal alignment. Extensive experiments conducted on the MAG datasets demonstrate that SMGFM achieves state-of-the-art performance across graph-level and modality-level tasks.

Problem

• multimodal-attributed graphs
• semantic roles
• graph-frequency variation
• cross-modal fusion
• modality-intrinsic semantics

Innovation

• spectral graph learning
• multimodal-attributed graphs
• graph frequency decomposition
• modality-specific semantics
• Chebyshev filters