Technology Category
Reasoning under Uncertainty: Relational Probabilistic ModelsData Mining & Knowledge Management: Intelligent Query ProcessingMachine Learning: Statistical Relational/Logic Learning
Extracting Document Relations from Search Corpus by Marginalizing over User Queries
🤖 AI Summary
This paper addresses unsupervised document relation extraction. We propose the EDR-MQ framework and the MC-RAG method, which— for the first time—automatically model the joint probability distribution over document pairs via query marginalization on large-scale search logs, eliminating the need for manual annotations or predefined relation schemas. Leveraging diverse user queries, our approach captures document co-occurrence patterns and explicitly models retrieval dependencies across multiple query contexts through a conditional retrieval-augmented generation mechanism. Experiments demonstrate that the method effectively identifies thematic clusters, evidentiary chains, and cross-domain semantic associations, substantially outperforming conventional similarity-based metrics. Crucially, under fully unsupervised settings, it uncovers deep information pathways and latent semantic structures, offering a novel paradigm for open-domain knowledge discovery.
Technology Category
Application Category
📝 Abstract
Understanding relationships between documents in large-scale corpora is essential for knowledge discovery and information organization. However, existing approaches rely heavily on manual annotation or predefined relationship taxonomies. We propose EDR-MQ (Extracting Document Relations by Marginalizing over User Queries), a novel framework that discovers document relationships through query marginalization. EDR-MQ is based on the insight that strongly related documents often co-occur in results across diverse user queries, enabling us to estimate joint probabilities between document pairs by marginalizing over a collection of queries. To enable this query marginalization approach, we develop Multiply Conditioned Retrieval-Augmented Generation (MC-RAG), which employs conditional retrieval where subsequent document retrievals depend on previously retrieved content. By observing co-occurrence patterns across diverse queries, EDR-MQ estimates joint probabilities between document pairs without requiring labeled training data or predefined taxonomies. Experimental results show that our query marginalization approach successfully identifies meaningful document relationships, revealing topical clusters, evidence chains, and cross-domain connections that are not apparent through traditional similarity-based methods. Our query-driven framework offers a practical approach to document organization that adapts to different user perspectives and information needs.
Problem
Research questions and friction points this paper is trying to address.
Discovering document relationships without manual annotation
Estimating joint probabilities via query marginalization
Identifying meaningful connections beyond similarity-based methods
Innovation
Methods, ideas, or system contributions that make the work stand out.
Extracts document relations via query marginalization
Uses Multiply Conditioned Retrieval-Augmented Generation
Identifies relationships without labeled data
🔎 Similar Papers
