From Relevance to Utility: Evidence Retrieval with Feedback for Fact Verification

2023-10-18

Conference on Empirical Methods in Natural Language Processing

AI Summary

Existing fact-checking retrieval models primarily rely on relevance ranking, neglecting the actual discriminative utility of retrieved evidence for verifiers. Method: We propose a “utility-driven” evidence retrieval paradigm that directly enhances the support of retrieved evidence for claim verification. To this end, we design a Feedback-based Evidence Retriever (FER), which employs the KL divergence between the verifier’s (e.g., BERT-based) prediction distributions over retrieved versus gold evidence as a differentiable feedback signal, enabling end-to-end joint optimization of retrieval and verification. Our approach integrates the retriever, verifier, and KL-based feedback mechanism without requiring human-annotated evidence. Contribution/Results: On benchmarks such as FEVER, FER significantly outperforms prevailing relevance-driven baselines. It is the first work to systematically demonstrate that utility-oriented retrieval yields substantial improvements in fact verification performance, offering a novel, interpretable, and efficient pathway for automated fact-checking.

Abstract

Retrieval-enhanced methods have become a primary approach in fact verification (FV); it requires reasoning over multiple retrieved pieces of evidence to verify the integrity of a claim. To retrieve evidence, existing work often employs off-the-shelf retrieval models whose design is based on the probability ranking principle. We argue that, rather than relevance, for FV we need to focus on the utility that a claim verifier derives from the retrieved evidence. We introduce the feedback-based evidence retriever(FER) that optimizes the evidence retrieval process by incorporating feedback from the claim verifier. As a feedback signal we use the divergence in utility between how effectively the verifier utilizes the retrieved evidence and the ground-truth evidence to produce the final claim label. Empirical studies demonstrate the superiority of FER over prevailing baselines.

Problem

• Optimizing evidence retrieval for fact verification
• Focusing on utility over relevance ranking
• Incorporating verifier feedback to improve retrieval

Innovation

• Feedback-based evidence retriever optimizes retrieval process
• Incorporates verifier utility divergence as feedback signal
• Focuses on utility over relevance for fact verification