Relation-Aware Graph Foundation Model

2025-05-17

AI Summary

Graph foundation models (GFMs) suffer from limited generalization due to the absence of a universal, token-like representational unit—akin to “tokens” in natural language—for graph structures. To address this, we propose modeling **relations as the fundamental unit**, enabling learnable relational vocabulary construction. We design a **tripartite conditional hypernetwork architecture** that dynamically generates: (i) relation-aware aggregators, (ii) task-adaptive classifiers, and (iii) dataset-adaptive projectors. Our framework integrates hybrid multi-source graph pretraining with dual structural and attribute augmentation. This is the first approach to achieve *tokenized representation learning* for graphs and *parameter-level cross-graph adaptation*. Evaluated across diverse heterogeneous graph benchmarks, it consistently outperforms state-of-the-art GFMs on downstream tasks, establishing new SOTA performance and demonstrating superior generalization and universality.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Reasoning under Uncertainty: Relational Probabilistic Models
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

In recent years, large language models (LLMs) have demonstrated remarkable generalization capabilities across various natural language processing (NLP) tasks. Similarly, graph foundation models (GFMs) have emerged as a promising direction in graph learning, aiming to generalize across diverse datasets through large-scale pre-training. However, unlike language models that rely on explicit token representations, graphs lack a well-defined unit for generalization, making it challenging to design effective pre-training strategies. In this work, we propose REEF, a novel framework that leverages relation tokens as the basic units for GFMs. Inspired by the token vocabulary in LLMs, we construct a relation vocabulary of relation tokens to store relational information within graphs. To accommodate diverse relations, we introduce two hypernetworks that adaptively generate the parameters of aggregators and classifiers in graph neural networks based on relation tokens. In addition, we design another hypernetwork to construct dataset-specific projectors and incorporate a dataset-level feature bias into the initial node representations, enhancing flexibility across different datasets with the same relation. Further, we adopt graph data augmentation and a mixed-dataset pre-training strategy, allowing REEF to capture relational diversity more effectively and exhibit strong generalization capabilities. Extensive experiments show that REEF significantly outperforms existing methods on both pre-training and transfer learning tasks, underscoring its potential as a powerful foundation model for graph-based applications.

Problem

• Lack of well-defined units for generalization in graph learning
• Challenges in designing effective pre-training strategies for graph models
• Need for flexible adaptation across diverse graph datasets and relations

Innovation

• Uses relation tokens as basic units for GFMs
• Introduces hypernetworks for adaptive parameter generation
• Employs graph data augmentation and mixed-dataset pre-training