A Systematic Investigation of Document Chunking Strategies and Embedding Sensitivity

2026-03-07

AI Summary

This study addresses the lack of systematic, cross-domain evaluation of document chunking strategies in dense retrieval systems, which has hindered the effectiveness of retrieval-augmented applications. For the first time, it presents a large-scale benchmark across six knowledge domains, evaluating 36 chunking methods—including fixed-length, semantic, structure-aware, hierarchical, adaptive, and LLM-assisted approaches—using five embedding models and fine-grained relevance annotations generated by large language models. The results demonstrate that content-aware chunking significantly outperforms fixed-length splitting, with optimal strategies varying by domain: paragraph-group chunking achieves the best overall performance (average nDCG@5 = 0.459), dynamic chunking excels in biology, physics, and health domains, while paragraph-level chunking is superior in legal and mathematical contexts. The work also quantifies the trade-off between retrieval effectiveness and computational efficiency, offering practical guidance for system design.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Natural Language Processing: Syntax — Tagging, Chunking & Parsing
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We present the first large-scale, cross-domain evaluation of document chunking strategies for dense retrieval, addressing a critical but underexplored aspect of retrieval-augmented systems. In our study, 36 segmentation methods spanning fixed-size, semantic, structure-aware, hierarchical, adaptive, and LLM-assisted approaches are benchmarked across six diverse knowledge domains using five different embedding models. Retrieval performance is assessed using graded relevance scores from a state-of-the-art LLM evaluator, with Normalised DCG@5 as the primary metric (complemented by Hit@5 and MRR). Our experiments show that content-aware chunking significantly improves retrieval effectiveness over naive fixed-length splitting. The top-performing strategy, Paragraph Group Chunking, achieved the highest overall accuracy (mean nDCG@5~0.459) and substantially better top-rank hit rates (Precision@1~24%, Hit@5~59%). In contrast, simple fixed-size character chunking as baselines performed poorly (nDCG@5<0.244, Precision@1~2-3%). We observe pronounced domain-specific differences: dynamic token sizing is strongest in biology, physics and health, while paragraph grouping is strongest in legal and maths. Larger embedding models yield higher absolute scores but remain sensitive to suboptimal segmentation, indicating that better chunking and large embeddings provide complementary benefits. In addition to accuracy gains, we quantify the efficiency trade-offs of advanced chunking. Producing more, smaller chunks can increase index size and latency. Consequently, we identify methods (like dynamic chunking) that approach an optimal balance of effectiveness and efficiency. These findings establish chunking as a vital lever for improving retrieval performance and reliability.

Problem

• document chunking
• dense retrieval
• retrieval-augmented systems
• embedding sensitivity
• segmentation strategies

Innovation

• document chunking
• dense retrieval
• cross-domain evaluation
• embedding sensitivity
• retrieval-augmented generation