Context-Aware Search and Retrieval Under Token Erasure

2026-04-20

AI Summary

This work addresses the reliability of retrieval-augmented generation (RAG) systems under query token erasure, which can lead to partial loss of query representations. To mitigate this issue, the authors propose a semantic importance-aware adaptive redundancy allocation mechanism that constructs query representations based on term frequency and retrieves documents using TF-IDF–weighted similarity. For the first time, the retrieval error probability is characterized from an information-theoretic perspective, yielding a multivariate Gaussian approximation of the similarity margin along with a computable upper bound. Both theoretical analysis and empirical experiments consistently demonstrate that allocating higher redundancy to semantically more important query features significantly enhances retrieval reliability—a principle that remains effective in embedding-based retrieval on real-world datasets.

Technology Category

• Data Mining & Knowledge Management: Intelligent Query Processing
• Reasoning under Uncertainty: Relational Probabilistic Models
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

This paper introduces and analyzes a search and retrieval model for RAG-like systems under {token} erasures. We provide an information-theoretic analysis of remote document retrieval when query representations are only partially preserved. The query is represented using term-frequency-based features, and semantically adaptive redundancy is assigned according to feature importance. Retrieval is performed using TF-IDF-weighted similarity. We characterize the retrieval error probability by showing that the vector of similarity margins converges to a multivariate Gaussian distribution, yielding an explicit approximation and computable upper bounds. Numerical results support the analysis, while a separate data-driven evaluation using embedding-based retrieval on real-world data shows that the same importance-aware redundancy principles extend to modern retrieval pipelines. Overall, the results show that assigning higher redundancy to semantically important query features improves retrieval reliability.

Problem

• token erasure
• context-aware retrieval
• query representation
• retrieval reliability
• RAG systems

Innovation

• token erasure
• semantic redundancy
• TF-IDF retrieval
• information-theoretic analysis
• importance-aware encoding