🤖 AI Summary
This work addresses the limitations of traditional document chunking methods, which often ignore user intent and consequently produce fragmented or irrelevant retrieved content, degrading retrieval performance. To overcome this, the authors propose an intent-aware dynamic chunking approach that first leverages a large language model to infer the underlying information needs behind user queries and then applies a dynamic programming algorithm to achieve globally optimal document segmentation, avoiding the suboptimal solutions typical of greedy strategies. As the first study to integrate dynamic programming into intent-driven chunking, the method significantly outperforms existing baselines across six question-answering datasets, improving top-1 retrieval accuracy by 5%–67%, reducing the number of chunks by 40%–60%, and maintaining answer coverage at 93%–100%.
📝 Abstract
Breaking long documents into smaller segments is a fundamental challenge in information retrieval. Whether for search engines, question-answering systems, or retrieval-augmented generation (RAG), effective segmentation determines how well systems can locate and return relevant information. However, traditional methods, such as fixed-length or coherence-based segmentation, ignore user intent, leading to chunks that split answers or contain irrelevant noise. We introduce Intent-Driven Dynamic Chunking (IDC), a novel approach that uses predicted user queries to guide document segmentation. IDC leverages a Large Language Model to generate likely user intents for a document and then employs a dynamic programming algorithm to find the globally optimal chunk boundaries. This represents a novel application of DP to intent-aware segmentation that avoids greedy pitfalls. We evaluated IDC on six diverse question-answering datasets, including news articles, Wikipedia, academic papers, and technical documentation. IDC outperformed traditional chunking strategies on five datasets, improving top-1 retrieval accuracy by 5% to 67%, and matched the best baseline on the sixth. Additionally, IDC produced 40-60% fewer chunks than baseline methods while achieving 93-100% answer coverage. These results demonstrate that aligning document structure with anticipated information needs significantly boosts retrieval performance, particularly for long and heterogeneous documents.