Retriever-and-Memory: Towards Adaptive Note-Enhanced Retrieval-Augmented Generation

2024-10-11

arXiv.org

AI Summary

Existing RAG methods for complex open-domain question answering rely on a single LLM invocation to determine retrieval timing and directly generate answers, failing to accurately model dynamic information needs and resulting in inefficient knowledge utilization. Method: We propose Note-RAG, an adaptive note-augmented RAG framework featuring a novel “retriever–memory” dual-module paradigm. It enables progressive knowledge construction through iterative retrieval and LLM-driven generation of structured notes; further, it introduces a note-state-aware adaptive memory review and exploration termination mechanism to improve retrieval timing decisions and knowledge interaction quality. Contribution/Results: Note-RAG achieves significant improvements over state-of-the-art RAG baselines across five challenging QA benchmarks. Ablation studies validate the effectiveness of each component. The code and datasets are publicly released.

Technology Category

• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Retrieval-Augmented Generation (RAG) mitigates issues of the factual errors and hallucinated outputs generated by Large Language Models (LLMs) in open-domain question-answering tasks (OpenQA) via introducing external knowledge. For complex QA, however, existing RAG methods use LLMs to actively predict retrieval timing and directly use the retrieved information for generation, regardless of whether the retrieval timing accurately reflects the actual information needs, or sufficiently considers prior retrieved knowledge, which may result in insufficient information gathering and interaction, yielding low-quality answers. To address these, we propose a generic RAG approach called Adaptive Note-Enhanced RAG (Adaptive-Note) for complex QA tasks, which includes the iterative information collector, adaptive memory reviewer, and task-oriented generator, while following a new Retriever-and-Memory paradigm. Specifically, Adaptive-Note introduces an overarching view of knowledge growth, iteratively gathering new information in the form of notes and updating them into the existing optimal knowledge structure, enhancing high-quality knowledge interactions. In addition, we employ an adaptive, note-based stop-exploration strategy to decide"what to retrieve and when to stop"to encourage sufficient knowledge exploration. We conduct extensive experiments on five complex QA datasets, and the results demonstrate the superiority and effectiveness of our method and its components. The code and data are at https://github.com/thunlp/Adaptive-Note.

Problem

• Mitigates factual errors in LLMs for QA using external knowledge
• Improves adaptive retrieval by refining knowledge through note-centric approach
• Enhances answer generation by leveraging high-quality accumulated notes

Innovation

• Note-centric adaptive retrieval framework
• Iterative knowledge growth assessment
• High-density quality knowledge gathering