CHARM: A Multimodal Graph Foundation Model with Hierarchical Context Modeling for Zero-Shot Transfer

2026-07-28

AI Summary

This work addresses the limitations of existing approaches in zero-shot transfer on multimodal graphs, which often rely on target-domain fine-tuning and struggle to effectively capture cross-modal associations. To overcome these challenges, the authors propose a zero-shot transfer framework that operates without any target-domain adaptation. The method leverages hierarchical graph contextual modeling to map nodes into a shared high-level semantic space and introduces a modality-aware graph encoder to generate graph tokens. These tokens are subsequently fed into a large language model to jointly integrate multimodal semantics and structural relationships. Experimental results demonstrate that the proposed approach significantly outperforms current state-of-the-art methods across multiple zero-shot multimodal graph tasks, confirming its strong generalization capability and effectiveness.

Abstract

Graph foundation models (GFMs) have emerged as a promising paradigm for transferring knowledge across graph domains and tasks. Real-world graphs associate nodes with text, images, and other modalities, making multimodal graphs essential for representing complex entities and relations. Moreover, collecting labels and adapting models for every new graph domain is costly and often infeasible, motivating zero-shot transfer. Unfortunately, zero-shot transfer on multimodal graphs remains underexplored. Existing GNN-based graph foundation models typically require downstream adaptation, whereas LLM-based graph methods mainly address unimodal graphs or tasks within a single domain. This setting presents two key challenges. First, models must generalize knowledge from individual modalities while capturing transferable cross-modal relations. Second, without target-domain fine-tuning, node representations remain entangled with domain-specific structures and modality-specific characteristics, obscuring shared concepts in unseen domains. To address these challenges, we propose CHARM, a multimodal graph foundation model with hierarchical context modeling for zero-shot transfer. CHARM replaces isolated raw nodes with hierarchical graph contexts that capture multimodal semantics and cross-modal relations. These contexts map domain-specific node patterns to shared high-level concepts, reducing reliance on target-domain supervision or adaptation. A modality-aware graph context encoder integrates multimodal information with graph structure and converts the resulting representations into graph tokens for a large language model . Experiments show consistent improvements on zero-shot multimodal graph tasks.

Problem

• multimodal graphs
• zero-shot transfer
• graph foundation models
• cross-modal relations
• domain generalization

Innovation

• multimodal graph
• zero-shot transfer
• hierarchical context modeling
• graph foundation model
• cross-modal relations