Graph Linearization Methods for Reasoning on Graphs with Large Language Models

2024-10-25

arXiv.org

AI Summary

Large language models (LLMs) lack native capability to process graph-structured data, hindering their application in graph reasoning tasks. Method: This paper proposes a semantic-driven graph linearization method that maps graphs into token sequences respecting natural language’s local dependencies and global alignment properties. We first formally define linearization principles balancing locality and global alignment. Then, we design an interpretable and generalizable node ordering strategy integrating PageRank, degree centrality, and k-core decomposition order, augmented by node relabeling for robustness. Finally, we introduce an end-to-end prompt-tuning and evaluation framework tailored for LLMs. Contribution/Results: Our method achieves significant improvements over random linearization baselines across multiple graph reasoning benchmarks. Empirical results demonstrate that the proposed representation effectively activates LLMs’ implicit graph cognition capabilities. This work establishes a novel paradigm for unifying graph and textual modeling within multimodal Transformer architectures.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Large language models have evolved to process multiple modalities beyond text, such as images and audio, which motivates us to explore how to effectively leverage them for graph reasoning tasks. The key question, therefore, is how to transform graphs into linear sequences of tokens, a process we term"graph linearization", so that LLMs can handle graphs naturally. We consider that graphs should be linearized meaningfully to reflect certain properties of natural language text, such as local dependency and global alignment, in order to ease contemporary LLMs, trained on trillions of textual tokens, better understand graphs. To achieve this, we developed several graph linearization methods based on graph centrality and degeneracy. These methods are further enhanced using node relabeling techniques. The experimental results demonstrate the effectiveness of our methods compared to the random linearization baseline. Our work introduces novel graph representations suitable for LLMs, contributing to the potential integration of graph machine learning with the trend of multimodal processing using a unified transformer model.

Problem

• Transform graphs into linear token sequences for LLM processing
• Enhance graph linearization to reflect natural language properties
• Develop methods based on centrality and degeneracy for better LLM integration

Innovation

• Graph linearization for LLM token processing
• Node relabeling enhances centrality-based methods
• Unified transformer for multimodal graph integration