Doc2Query++: Topic-Coverage based Document Expansion and its Application to Dense Retrieval via Dual-Index Fusion

2025-10-10

AI Summary

Document expansion (DE) in sparse retrieval suffers from hallucination, redundancy, low diversity, poor cross-domain generalization, and indexing noise that degrades dense retrieval—largely due to uncontrolled LLM-generated expansions. To address these issues, this paper proposes Topic-guided Dual-index DE, a novel method that integrates unsupervised topic modeling with hybrid keyword filtering to explicitly constrain LLMs to generate diverse yet semantically coherent queries. A dual-index fusion mechanism is designed to separately index original documents and expanded queries, thereby eliminating concatenation-induced noise. Evaluated on cross-domain benchmarks including BEIR, our approach significantly improves both sparse and dense retrieval performance, outperforming state-of-the-art methods across MAP, nDCG@10, and Recall@100. To the best of our knowledge, this is the first work to jointly achieve high-quality, controllable generation and robust cross-domain retrieval.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Distributed Search

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Web Mining and Content Analysis: Topic discovery and tracking

Abstract

Document expansion (DE) via query generation tackles vocabulary mismatch in sparse retrieval, yet faces limitations: uncontrolled generation producing hallucinated or redundant queries with low diversity; poor generalization from in-domain training (e.g., MS MARCO) to out-of-domain data like BEIR; and noise from concatenation harming dense retrieval. While Large Language Models (LLMs) enable cross-domain query generation, basic prompting lacks control, and taxonomy-based methods rely on domain-specific structures, limiting applicability. To address these challenges, we introduce Doc2Query++, a DE framework that structures query generation by first inferring a document's latent topics via unsupervised topic modeling for cross-domain applicability, then using hybrid keyword selection to create a diverse and relevant keyword set per document. This guides LLM not only to leverage keywords, which ensure comprehensive topic representation, but also to reduce redundancy through diverse, relevant terms. To prevent noise from query appending in dense retrieval, we propose Dual-Index Fusion strategy that isolates text and query signals, boosting performance in dense settings. Extensive experiments show Doc2Query++ significantly outperforms state-of-the-art baselines, achieving substantial gains in MAP, nDCG@10 and Recall@100 across diverse datasets on both sparse and dense retrieval.

Problem

• Addresses uncontrolled query generation causing redundancy in document expansion
• Improves cross-domain generalization for dense and sparse retrieval systems
• Reduces noise from query concatenation via dual-index fusion strategy

Innovation

• Unsupervised topic modeling guides cross-domain query generation
• Hybrid keyword selection ensures diverse and relevant query terms
• Dual-Index Fusion isolates text and query signals separately