🤖 AI Summary
This work proposes a novel approach to open-domain claim verification that addresses the limitations of existing fact-checking systems, which often rely on a single knowledge source and thus fail to capture perspective divergence, resulting in limited coverage and poor transparency. The method leverages large language models (LLMs) to simultaneously retrieve multi-source evidence—such as from Wikipedia, PubMed, and Google—for both the original claim and its negation, integrating supporting and contradicting information. It further incorporates cross-source disagreement analysis to better model the complexity and diversity of information. By combining evidence deduplication, confidence scoring, and visualization, the approach is evaluated across four benchmark datasets using five distinct LLMs, achieving significant accuracy improvements and revealing notable differences in how various knowledge sources contribute to reasoning.
📝 Abstract
The spread of misinformation across digital platforms can pose significant societal risks. Claim verification, a.k.a. fact-checking, systems can help identify potential misinformation. However, their efficacy is limited by the knowledge sources that they rely on. Most automated claim verification systems depend on a single knowledge source and utilize the supporting evidence from that source; they ignore the disagreement of their source with others. This limits their knowledge coverage and transparency. To address these limitations, we present a novel system for open-domain claim verification (ODCV) that leverages large language models (LLMs), multi-perspective evidence retrieval, and cross-source disagreement analysis. Our approach introduces a novel retrieval strategy that collects evidence for both the original and the negated forms of a claim, enabling the system to capture supporting and contradicting information from diverse sources: Wikipedia, PubMed, and Google. These evidence sets are filtered, deduplicated, and aggregated across sources to form a unified and enriched knowledge base that better reflects the complexity of real-world information. This aggregated evidence is then used for claim verification using LLMs. We further enhance interpretability by analyzing model confidence scores to quantify and visualize inter-source disagreement. Through extensive evaluation on four benchmark datasets with five LLMs, we show that knowledge aggregation not only improves claim verification but also reveals differences in source-specific reasoning. Our findings underscore the importance of embracing diversity, contradiction, and aggregation in evidence for building reliable and transparent claim verification systems