Knowledge-Instruct: Effective Continual Pre-training from Limited Data using Instructions

2025-04-08

AI Summary

To address the challenges of inefficient knowledge injection, severe catastrophic forgetting, and degradation of general capabilities in low-resource continual pretraining (CPT) of large language models (LLMs), this paper proposes a novel knowledge injection paradigm based solely on instruction tuning. Methodologically, it pioneers the use of small models to synthesize high-information-density, multi-hop reasoning–enhanced instruction data for targeted knowledge updates; further, it integrates knowledge distillation with retrieval-augmented modeling to jointly strengthen factual memory retention and preserve general reasoning and instruction-following abilities. A dedicated benchmark—Companies—is introduced to rigorously evaluate knowledge injection efficacy. Experimental results demonstrate that our approach significantly outperforms conventional CPT under extremely limited data budgets, achieving breakthroughs in mitigating forgetting, improving factual accuracy, enhancing complex contextual understanding, and enabling robust retrieval integration.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: (Large) Language ModelsCognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationSearch and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

While Large Language Models (LLMs) acquire vast knowledge during pre-training, they often lack domain-specific, new, or niche information. Continual pre-training (CPT) attempts to address this gap but suffers from catastrophic forgetting and inefficiencies in low-data regimes. We introduce Knowledge-Instruct, a novel approach to efficiently inject knowledge from limited corpora through pure instruction-tuning. By generating information-dense synthetic instruction data, it effectively integrates new knowledge while preserving general reasoning and instruction-following abilities. Knowledge-Instruct demonstrates superior factual memorization, minimizes catastrophic forgetting, and remains scalable by leveraging synthetic data from relatively small language models. Additionally, it enhances contextual understanding, including complex multi-hop reasoning, facilitating integration with retrieval systems. We validate its effectiveness across diverse benchmarks, including Companies, a new dataset that we release to measure knowledge injection capabilities.

Problem

• Addresses lack of domain-specific knowledge in LLMs
• Reduces catastrophic forgetting in continual pre-training
• Improves knowledge injection from limited data

Innovation

• Uses instruction-tuning for knowledge injection
• Generates synthetic data to prevent forgetting
• Enhances contextual and multi-hop reasoning