🤖 AI Summary
This work addresses the misalignment between graph representations and the latent feature space of frozen large language models in graph retrieval-augmented generation. To bridge this gap, the authors propose AGE, a method built upon a masked self-supervised learning framework that encodes graphs using a Transformer architecture with text-like embeddings. A key innovation of AGE is its learnable node sampler, which adaptively identifies and bypasses hard-to-predict yet critical nodes to better align graph and textual embedding spaces. Evaluated on four heterogeneous GraphQA benchmark datasets, AGE substantially outperforms existing non-parametric retrieval approaches, achieving state-of-the-art accuracy.
📝 Abstract
GraphRAG is an extension of retrieval-augmented generation (RAG) that supports large language models (LLMs) by referring to graph-structured data as external knowledge. While this technique ideally captures intricate relationships, it often struggles with graph representations for LLMs, particularly for frozen LLMs, due to the misalignment between graph-based and text-based latent features. We tackle this issue by introducing the {\it Adaptive-masking for Graph Embedding (AGE)}. AGE employs a Transformer in a mask-based self-supervised learning (SSL) approach. We designed the architecture similar to text embedding encoders, addressing the latent feature misalignment. In contrast to natural language texts, graphs are concise representations, and there exist {\it key nodes} that hold dominant contextual information, which are challenging to predict from their surroundings. Masking such key nodes leads to inefficiency in the SSL process. Therefore, AGE focuses on predicting nodes apart from key nodes, utilizing a learnable node sampler. Our experimental results indicate that AGE significantly improves approaches using non-parametric search component in GraphQA tasks, achieving superior accuracy across four benchmark datasets with distinct characteristics.