M-RAG: Making RAG Faster, Stronger, and More Efficient

2026-01-06

arXiv.org

AI Summary

This work addresses a key limitation of traditional retrieval-augmented generation (RAG) approaches, wherein tight coupling of retrieval and generation within fixed text chunks leads to either information fragmentation from overly fine-grained segmentation or excessive noise and inefficient context budget usage from overly large chunks. To overcome this, the authors propose M-RAG, which introduces a novel semantic key-value indexing structure that decouples retrieval from generation. Documents are parsed into records comprising retrieval keys, informative values, and source pointers; retrieval operates over keys—supporting both dense and sparse methods—while generation dynamically assembles corresponding values within token budget constraints. Evaluated on LongBench question-answering tasks, M-RAG matches or outperforms leading chunk-based baselines, particularly under strict token limits, while achieving high document coverage, strong robustness, and reduced online retrieval latency.

Abstract

Retrieval-Augmented Generation (RAG) has become a widely adopted paradigm for enhancing the reliability of large language models (LLMs). However, RAG systems are sensitive to retrieval strategies that rely on text chunking to construct retrieval units, which often introduce information fragmentation, retrieval noise, and reduced efficiency. Recent work has even questioned the necessity of RAG, arguing that long-context LLMs may eliminate multi-stage retrieval pipelines by directly processing full documents. Nevertheless, expanded context capacity alone does not resolve the challenges of relevance filtering, evidence prioritization, and isolating answer-bearing information. To this end, we proposed M-RAG, a novel Chunk-free retrieval strategy. Instead of retrieving coarse-grained textual chunks, M-RAG extracts structured, k-v decomposition meta-markers, with a lightweight, intent-aligned retrieval key for retrieval and a context-rich information value for generation. Under this setting, M-RAG enables efficient and stable query-key similarity matching without sacrificing expressive ability. Experimental results on the LongBench subtasks demonstrate that M-RAG outperforms chunk-based RAG baselines across varying token budgets, particularly under low-resource settings. Extensive analysis further reveals that M-RAG retrieves more answer-friendly evidence with high efficiency, validating the effectiveness of decoupling retrieval representation from generation and highlighting the proposed strategy as a scalable and robust alternative to existing chunk-based methods.

Problem

• Retrieval-Augmented Generation
• semantic indexing
• token budget
• key-value indexing
• evidence retrieval

Innovation

• semantic key-value indexing
• retrieval-augmented generation
• token budget
• evidence payload
• provenance pointers