Multi-Agent Fact Checking

2025-03-03

AI Summary

This paper addresses fake news detection under multi-source agents with unknown reliability. Each agent provides binary labels for news items with heterogeneous error rates πᵢ, and agent reliability is initially unknown and must be learned online. Methodologically, we propose a distributed online fact-checking framework that models agent unreliability as individualized error parameters and develops an online learning algorithm integrating stochastic approximation with distributed consensus. We rigorously prove consistency of parameter estimation and convergence in discrete time. Theoretically, our approach enables adaptive decision-making without requiring prior trust assessments. Empirical evaluations demonstrate that the method significantly improves overall accuracy in settings involving heterogeneous, multi-source agents.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Multiagent Systems: Adversarial Agents
• Machine Learning: Online Learning & Bandits

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity

Abstract

We formulate the problem of fake news detection using distributed fact-checkers (agents) with unknown reliability. The stream of news/statements is modeled as an independent and identically distributed binary source (to represent true and false statements). Upon observing a news, agent $i$ labels the news as true or false which reflects the true validity of the statement with some probability $1-pi_i$. In other words, agent $i$ misclassified each statement with error probability $pi_iin (0,1)$, where the parameter $pi_i$ models the (un)trustworthiness of agent $i$. We present an algorithm to learn the unreliability parameters, resulting in a distributed fact-checking algorithm. Furthermore, we extensively analyze the discrete-time limit of our algorithm.

Problem

• Detecting fake news using distributed agents with unknown reliability.
• Modeling news streams as binary sources for true/false statements.
• Learning agent unreliability parameters for improved fact-checking.

Innovation

• Distributed agents detect fake news
• Algorithm learns agent unreliability parameters
• Analyzes discrete-time limit of algorithm