SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding

2024-08-28

arXiv.org

AI Summary

Large language models (LLMs) face dual challenges in scientific literature understanding: insufficient domain-specific knowledge and poor task alignment. To address these, we propose a “knowledge injection–task alignment” collaborative adaptation framework. Our method introduces a novel scientific text quality enhancement pipeline and constructs SciLitIns—the first high-quality instruction dataset tailored to niche scientific domains—generated via an LLM-driven synthetic instruction approach. The pipeline integrates robust PDF parsing, multi-stage quality filtering, continued pretraining (CPT), and supervised fine-tuning (SFT). The resulting model, SciLitLLM, achieves significant performance gains over general-purpose baselines across multiple scientific literature understanding benchmarks, empirically validating the efficacy of synergistic knowledge enhancement and task-specific refinement. Moreover, the framework demonstrates cross-domain transferability, establishing a systematic paradigm for adapting foundation models to specialized scientific domains.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Application Domains: Natural Sciences

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Scientific literature understanding is crucial for extracting targeted information and garnering insights, thereby significantly advancing scientific discovery. Despite the remarkable success of Large Language Models (LLMs), they face challenges in scientific literature understanding, primarily due to (1) a lack of scientific knowledge and (2) unfamiliarity with specialized scientific tasks. To develop an LLM specialized in scientific literature understanding, we propose a hybrid strategy that integrates continual pre-training (CPT) and supervised fine-tuning (SFT), to simultaneously infuse scientific domain knowledge and enhance instruction-following capabilities for domain-specific tasks.cIn this process, we identify two key challenges: (1) constructing high-quality CPT corpora, and (2) generating diverse SFT instructions. We address these challenges through a meticulous pipeline, including PDF text extraction, parsing content error correction, quality filtering, and synthetic instruction creation. Applying this strategy, we present a suite of LLMs: SciLitLLM, specialized in scientific literature understanding. These models demonstrate promising performance on scientific literature understanding benchmarks. Our contributions are threefold: (1) We present an effective framework that integrates CPT and SFT to adapt LLMs to scientific literature understanding, which can also be easily adapted to other domains. (2) We propose an LLM-based synthesis method to generate diverse and high-quality scientific instructions, resulting in a new instruction set -- SciLitIns -- for supervised fine-tuning in less-represented scientific domains. (3) SciLitLLM achieves promising performance improvements on scientific literature understanding benchmarks.

Problem

• Adapt LLMs for scientific literature understanding
• Enhance LLMs' scientific domain knowledge
• Improve LLMs' performance on specialized scientific tasks

Innovation

• Hybrid CPT-SFT strategy
• High-quality corpus construction
• Diverse instruction synthesis