ECoRAG: Evidentiality-guided Compression for Long Context RAG

2025-06-05

AI Summary

To address document redundancy and non-evidential interference in long-context retrieval-augmented generation (RAG) for open-domain question answering (ODQA), this paper proposes an evidence-driven document compression framework. Methodologically, it shifts from conventional “information preservation” to “evidence sufficiency” as the primary objective, integrating an evidence identification classifier, semantic importance re-ranking, adaptive truncation, and an LLM confidence feedback loop that enables retrieval–compression co-optimization—triggering incremental retrieval when evidence is insufficient. The core contributions are the first evidence-aware compression mechanism and an evidence-sufficiency feedback loop. Experiments demonstrate that our approach reduces token consumption and latency by over 30% compared to state-of-the-art compression methods, while simultaneously improving answer accuracy—significantly enhancing the cost-effectiveness of RAG systems.

Technology Category

• Data Mining & Knowledge Management: Data Compression
• Natural Language Processing: Question Answering
• Machine Learning: Learning on the Edge & Model Compression

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Community question answering

Abstract

Large Language Models (LLMs) have shown remarkable performance in Open-Domain Question Answering (ODQA) by leveraging external documents through Retrieval-Augmented Generation (RAG). To reduce RAG overhead, from longer context, context compression is necessary. However, prior compression methods do not focus on filtering out non-evidential information, which limit the performance in LLM-based RAG. We thus propose Evidentiality-guided RAG, or ECoRAG framework. ECoRAG improves LLM performance by compressing retrieved documents based on evidentiality, ensuring whether answer generation is supported by the correct evidence. As an additional step, ECoRAG reflects whether the compressed content provides sufficient evidence, and if not, retrieves more until sufficient. Experiments show that ECoRAG improves LLM performance on ODQA tasks, outperforming existing compression methods. Furthermore, ECoRAG is highly cost-efficient, as it not only reduces latency but also minimizes token usage by retaining only the necessary information to generate the correct answer. Code is available at https://github.com/ldilab/ECoRAG.

Problem

• Filter non-evidential information in RAG compression
• Ensure answer generation is evidence-supported
• Reduce latency and token usage efficiently

Innovation

• Evidentiality-guided document compression for RAG
• Dynamic retrieval until sufficient evidence is found
• Cost-efficient token usage by retaining essential information