LitFM: A Retrieval Augmented Structure-aware Foundation Model For Citation Graphs

2024-09-05

Knowledge Discovery and Data Mining

AI Summary

Existing large language models (LLMs) for scientific literature management neglect citation graph structure and fine-grained semantic relationships, leading to inaccurate relation identification, frequent hallucinations, poor generalization, and fragmented task adaptation. Method: We propose the first domain-specific, structure-aware foundation model for scholarly literature, featuring a novel dual-path architecture that integrates citation graph retrieval with knowledge-injected LLMs. Our approach incorporates graph-augmented retrieval, citation-aware graph encoding, knowledge-injected instruction tuning, and a unified joint training/inference framework. Contribution/Results: We introduce three new benchmark datasets with sentence-level citation annotations across distinct scientific domains. Our model achieves the first cross-task generalization and zero-shot relational reasoning over unseen papers. Experiments demonstrate a 28.1% improvement in citation retrieval accuracy and an average 7.52% gain over state-of-the-art methods across six downstream tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

With the advent of large language models (LLMs), managing scientific literature via LLMs has become a promising direction of research. However, existing approaches often overlook the rich structural and semantic relevance among scientific literature, limiting their ability to discern the relationships between pieces of scientific knowledge, and suffer from various types of hallucinations. These methods also focus narrowly on individual downstream tasks, limiting their applicability across use cases. We propose LitFM, the first literature foundation model designed for a wide variety of practical downstream tasks on domain-specific literature, with a focus on citation information. At its core, LitFM contains a novel graph retriever that can provide accurate and diverse recommendations for LLM to integrate graph structure information and relevant literature. LitFM also leverages a knowledge-infused LLM, fine-tuned through a well-developed instruction paradigm. It enables LitFM to extract domain-specific knowledge from literature and reason relationships among them. By integrating citation graphs during both training and inference, LitFM can generalize to unseen papers and accurately assess their relevance within existing literature. Additionally, we introduce new large-scale literature citation benchmark datasets on three academic fields, featuring sentence-level citation information and local context. Extensive experiments validate the superiority of LitFM, achieving 28.1% improvement on retrieval task in precision, and an average improvement of 7.52% over state-of-the-art across six downstream literature-related tasks.

Problem

• Existing methods overlook structural relationships in scientific literature
• Current approaches suffer from hallucinations and limited task applicability
• Models lack ability to generalize to unseen papers accurately

Innovation

• Graph retriever integrates citation structure for relevance
• Knowledge-infused LLM extracts and reasons domain-specific relationships
• Training with citation graphs generalizes to unseen papers