How to Make LLMs Strong Node Classifiers?

2024-10-03

AI Summary

How can general-purpose large language models (LLMs) match the performance of specialized graph neural networks (GNNs) on node classification? This paper proposes a lightweight, architecture-agnostic enhancement framework: it augments LLM inputs via dual-path retrieval—leveraging both topological neighborhood and semantic similarity—and incorporates a lightweight GNN as a learnable module for candidate pruning and classification guidance. Additionally, we introduce a multi-task joint instruction-tuning strategy that unifies node classification, retrieval, and structural understanding. Evaluated on multiple real-world graph benchmarks, our approach is the first to enable open-source LLMs (e.g., Flan-T5) to surpass state-of-the-art text-based classifiers under a text-output paradigm, while approaching the performance of top-performing vector-output GNNs—significantly narrowing the performance gap between general-purpose LLMs and domain-specific graph models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Language Models (LMs) are increasingly challenging the dominance of domain-specific models, such as Graph Neural Networks (GNNs) and Graph Transformers (GTs), in graph learning tasks. Following this trend, we propose a novel approach that empowers off-the-shelf LMs to achieve performance comparable to state-of-the-art (SOTA) GNNs on node classification tasks, without requiring any architectural modification. By preserving the LM's original architecture, our approach retains a key benefit of LM instruction tuning: the ability to jointly train on diverse datasets, fostering greater flexibility and efficiency. To achieve this, we introduce two key augmentation strategies: (1) Enriching LMs' input using topological and semantic retrieval methods, which provide richer contextual information, and (2) guiding the LMs' classification process through a lightweight GNN classifier that effectively prunes class candidates. Our experiments on real-world datasets show that backbone Flan-T5 LMs equipped with these augmentation strategies outperform SOTA text-output node classifiers and are comparable to top-performing vector-output node classifiers. By bridging the gap between specialized node classifiers and general LMs, this work paves the way for more versatile and widely applicable graph learning models. We will open-source the code upon publication.

Problem

• Large Language Models
• Node Classification
• Graph Neural Networks

Innovation

• Graph Learning Enhancement
• Auxiliary Graph Neural Network
• Language Model Optimization