Decomposition Dilemmas: Does Claim Decomposition Boost or Burden Fact-Checking Performance?

2024-10-17

arXiv.org

AI Summary

This work investigates whether claim decomposition improves fact-checking performance—a critical yet unresolved question. To address inconsistent findings in prior research, the authors construct the first taxonomy of decomposition errors and conduct systematic empirical studies across mainstream models and decomposers, employing error-case analysis, multi-round controlled experiments, and performance attribution. Results demonstrate that decomposition is not universally beneficial: its efficacy critically depends on input characteristics. While accuracy improves in certain scenarios, it degrades significantly in many others due to semantic distortion, information loss, or redundancy introduced during decomposition. This study is the first to reveal the intrinsic instability of the “decompose-then-verify” paradigm, highlighting the absence of controllability and robustness guarantees for decomposition quality in current approaches. The findings provide both theoretical grounding and practical warnings for optimizing fact-checking pipelines.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity

Abstract

Fact-checking pipelines increasingly adopt the Decompose-Then-Verify paradigm, where texts are broken down into smaller claims for individual verification and subsequently combined for a veracity decision. While decomposition is widely-adopted in such pipelines, its effects on final fact-checking performance remain underexplored. Some studies have reported improvements from decompostition, while others have observed performance declines, indicating its inconsistent impact. To date, no comprehensive analysis has been conducted to understand this variability. To address this gap, we present an in-depth analysis that explicitly examines the impact of decomposition on downstream verification performance. Through error case inspection and experiments, we introduce a categorization of decomposition errors and reveal a trade-off between accuracy gains and the noise introduced through decomposition. Our analysis provides new insights into understanding current system's instability and offers guidance for future studies toward improving claim decomposition in fact-checking pipelines.

Problem

• Impact of claim decomposition on fact-checking
• Variability in decomposition effects on performance
• Trade-off between accuracy and decomposition noise

Innovation

• Decompose-Then-Verify paradigm
• categorization of decomposition errors
• trade-off between accuracy and noise