Augmenting LLM Reasoning with Dynamic Notes Writing for Complex QA

2025-05-22

AI Summary

To address context redundancy, information decay, and limited long-range reasoning in iterative retrieval-augmented generation (RAG) for multi-hop question answering, this paper proposes a dynamic note-writing mechanism. After each retrieval step, a large language model (LLM) autonomously generates a concise, highly relevant core summary—requiring no fine-tuning and operating independently of the underlying RAG framework. This enables continuous context denoising and faithful preservation of critical information across iterations, overcoming the limitations of single-step compression and effectively extending the LLM’s usable context length. Leveraging controllable summarization and token-efficiency optimization, our method achieves an average 15.6-percentage-point improvement across two LLMs, four benchmark datasets, and three iterative RAG paradigms, with negligible increase in output token count. The approach demonstrates strong generalizability and practical utility.

Technology Category

• Natural Language Processing: Summarization
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Iterative RAG for multi-hop question answering faces challenges with lengthy contexts and the buildup of irrelevant information. This hinders a model's capacity to process and reason over retrieved content and limits performance. While recent methods focus on compressing retrieved information, they are either restricted to single-round RAG, require finetuning or lack scalability in iterative RAG. To address these challenges, we propose Notes Writing, a method that generates concise and relevant notes from retrieved documents at each step, thereby reducing noise and retaining only essential information. This indirectly increases the effective context length of Large Language Models (LLMs), enabling them to reason and plan more effectively while processing larger volumes of input text. Notes Writing is framework agnostic and can be integrated with different iterative RAG methods. We demonstrate its effectiveness with three iterative RAG methods, across two models and four evaluation datasets. Notes writing yields an average improvement of 15.6 percentage points overall, with minimal increase in output tokens.

Problem

• Addresses challenges in multi-hop QA with lengthy contexts
• Reduces irrelevant information buildup in iterative RAG
• Enhances LLM reasoning by generating concise dynamic notes

Innovation

• Dynamic notes writing reduces noise in RAG
• Enhances LLM reasoning with concise information
• Framework agnostic, integrates with iterative RAG