Language Models Benefit from Preparation with Elicited Knowledge

2024-09-02

arXiv.org

AI Summary

In knowledge-dependent question answering, zero-shot chain-of-thought (CoT) prompting overemphasizes reasoning trace generation while neglecting explicit knowledge acquisition. Method: This paper proposes PREP, a two-stage prompting framework: (1) a language model (LM) actively pre-extracts problem-relevant knowledge, and (2) a second LM generates the answer conditioned on this extracted knowledge. Contribution/Results: PREP introduces the first “knowledge extraction–reasoning” decoupled dual-LM collaborative prompting paradigm, requiring no domain-specific prompt engineering and exhibiting strong generalization. Evaluated on a newly constructed component–material dataset and three public commonsense reasoning benchmarks under zero-shot settings, PREP significantly outperforms zero-shot CoT and other baselines, achieving consistent average accuracy gains. Results empirically validate that explicit knowledge pre-extraction effectively enhances both instruction following and knowledge retrieval capabilities.

Technology Category

• Knowledge Representation and Reasoning: Common-Sense Reasoning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The zero-shot chain of thought (CoT) approach is often used in question answering (QA) by language models (LMs) for tasks that require multiple reasoning steps. However, some QA tasks hinge more on accessing relevant knowledge than on chaining reasoning steps. We introduce a simple prompting technique, called PREP, that involves using two instances of LMs: the first (LM1) generates relevant information, and the second (LM2) receives the information from the user and answers the question. This design is intended to make better use of the LM's instruction-following capability. PREP is applicable across various QA tasks without domain-specific prompt engineering. PREP is developed on a dataset of 100 QA questions, derived from an extensive schematic dataset specifying artifact parts and material composition. These questions ask which of two artifacts is less likely to share materials with another artifact. Such questions probe the LM's knowledge of shared materials in the part structure of different artifacts. We test our method on our parts-and-materials dataset and three published commonsense reasoning datasets. The average accuracy of our method is consistently higher than that of all the other tested methods across all the tested datasets.

Problem

• Improves QA by eliciting knowledge before reasoning
• Enhances LM performance without domain-specific prompts
• Tests accuracy on material composition and commonsense datasets

Innovation

• Uses two LMs for knowledge generation and answering
• PREP technique avoids domain-specific prompt engineering
• Improves accuracy on parts-and-materials QA tasks