Cross-Attention Calibrated Deduplication for Retrieval-Augmented Generation System

2026-07-27

AI Summary

This work addresses the inefficiencies in traditional retrieval-augmented generation (RAG) systems caused by redundant text chunks, which inflate vector databases and degrade retrieval performance. Existing deduplication methods often fail to distinguish between semantically similar and truly redundant content. To overcome this limitation, the authors propose a semantic deduplication approach based on cross-attention mechanisms. Their method replaces single-vector representations with cross-encoders to preserve token-level detail, introduces a novel information score (NIS) grounded in attention entropy to quantify redundancy, and incorporates a multi-candidate majority voting scheme to enhance robustness. Evaluated on SQuAD 1.1, the approach removes an average of 9.75% of redundant chunks while achieving a 27% speedup over the strongest baseline and operating approximately seven times faster than cosine similarity–based filtering.

Abstract

Common chunking strategies in Retrieval-Augmented Generation (RAG) systems often create redundant chunks. These redundant chunks make the vector database bigger and slow down retrieval. A common fix is cosine-similarity thresholding. This method reduces each chunk to a single vector, then compares vectors using a similarity score. But a single vector can lose the fine-grained, token-level detail needed to tell a true duplicate apart from a chunk that just shares the same topic. We propose Cross-Attention Calibrated Deduplication (CACD). CACD checks each new chunk against an in-memory pool of chunks already kept, using a cross-encoder instead of a single pooled vector. This keeps token-level detail all the way to the final comparison. CACD combines three parts: the cross-encoder comparison itself, a New Information Score (NIS) that measures how much of a chunk is not explained by a candidate already kept, and a majority vote across several candidates rather than a single best match. NIS is calculated from the attention entropy of the cross-encoder. We tested CACD against five existing filtering methods, nine chunking strategies, and 18 configurations, all on the full SQuAD 1.1 validation set. In our experiments, CACD removes 9.75% of chunks on average. This drop rate is close to other semantic-level methods, and much higher than exact-match filters, which barely remove anything. In these experiments, CACD also processes each configuration in 51.0 seconds on average, about 27% faster than the strongest baseline, NERExact (69.6s), and about 7x faster than cosine-similarity filtering (356.7s). These results come from a single dataset, so we present them as an early comparison, not a general claim. Code for the baseline evaluation and for CACD is available at https://github.com/lehuyphuong/rag_bench and https://github.com/lehuyphuong/cacd_dedup.

Problem

• Retrieval-Augmented Generation
• deduplication
• redundant chunks
• cross-attention
• semantic similarity

Innovation

• Cross-Attention
• Deduplication
• Retrieval-Augmented Generation
• New Information Score
• Cross-Encoder