🤖 AI Summary
This work addresses limitations in existing compound claim decomposition methods for automated fact-checking, which rely on lexical overlap metrics like Jaccard similarity and thus struggle to accurately assess the semantic fidelity of paraphrased atomic claims, while also lacking theoretical guarantees on the termination of repair processes. To overcome these issues, the authors propose CREDENCE, a framework that replaces lexical overlap with cosine similarity based on BGE-large embeddings to enable semantic-aware decomposition and self-repair. They formally prove, for the first time, the convergence of a hybrid repair pipeline combining symbolic rules and large language models. The study introduces Semantic-F1, a new evaluation metric validated across three cross-domain benchmarks—social media, encyclopedic texts, and news—demonstrating 15–32 percentage point improvements over Jaccard-F1, EPR scores of 0.94–1.00, and a 47%–100% reduction in atomicity violations via rule-based repair without compromising semantic fidelity.
📝 Abstract
Decomposing compound sentences into atomic, verifiable claims is a prerequisite for reliable automated fact-checking. Prior work has relied on token-overlap (Jaccard) metrics that systematically underestimate decomposition quality for paraphrastic claims, and has lacked formal termination analysis for the repair loop. We present Credence, a revised claim decomposition and evaluation framework addressing both shortcomings. Our contributions are: (1) Semantic-F1: we use BGE-large cosine similarity fidelity metric that resolves Jaccard's penalisation and improves downstream fact-checking accuracy; (2) Convergence theorems: we formally characterise four properties of the repair pipeline, establishing that rule-based repair is monotone and finitely terminating under an oracle parser assumption; LLM-based self-repair is provably non-monotone and requires an early-exit guard; (3) Three evaluation benchmarks spanning social-media, encyclopaedic, and news domains for cross-domain generalisation measurement; (4) Multi-model benchmarking across four decomposer models (3.8B-12B) and a closed API model. Experiments on SocialClaimSplit, WikiSplitBench, and ClaimDecompBench show that Semantic-F1 outperforms Jaccard-F1 by +15-32pp. EPR ranges from 0.94 to 1.00 on SocialClaimSplit and WikiSplitBench, while ClaimDecompBench includes lower base EPR cases (down to 0.824) due to harder news-domain constructions, and rule-repair reduces the Atomicity Violation Rate (AVR) by 47-100% relative to the base model without degrading fidelity.