On the Effectiveness of Fact Checking Information from Politically Congruent and Incongruent Large Language Models

2026-07-16

AI Summary

This study investigates how the political alignment of large language models (LLMs) influences users’ trust judgments in fact-checking contexts. Through two within-subjects experiments (N=705), it systematically examines, for the first time, how ideologically configured LLM chatbots differentially affect the correction of misinformation depending on the political distance between users and news content. Findings indicate that LLMs can effectively reduce user trust in false news even when their political stance is incongruent with the user’s; however, inaccurate or ambiguous fact-checks significantly undermine trust in true news, particularly when the political distance is greater. The research thus reveals both the potential and the risks of deploying LLM-based fact-checkers in enhancing the perceived credibility of truthful information.

Abstract

Social media companies have shifted away from human fact-checkers and instead have embedded conversational Large Language Models (LLM) on their platforms. LLM chatbots differ from human fact-checkers in many ways that may shape user responses to corrections. Of particular interest in this study is that LLM chatbots can be ideologically configured via the content emphasized in their responses, the sources cited, and the configured persona. Using data from two within-subjects experiments (n=705), this paper investigates the effectiveness of fact checking information from ideologically configured LLM chatbots. We find that LLM fact-checkers significantly shift trust in true and false political news headlines, even when the chatbot is politically incongruent with the user. The perceived political congruency between the participant and the bot matters only when headlines are politically distant. That is, trust in correctly labeled true headlines increases less when politically distant chatbots check distant headlines and increases more when moderate chatbots check distant headlines. The perceived political congruency of LLM chatbots did not impact their effectiveness at decreasing trust in false headlines. Unfortunately, LLM fact-checkers also significantly change trust in news when they are wrong or provide inconclusive answers. Our results demonstrate both the potential for LLMs to correct false information at scale but also their potential to taint the truth at scale.

Problem

• fact checking
• large language models
• political congruency
• misinformation
• trust

Innovation

• ideologically configured LLM
• fact-checking effectiveness
• political congruency
• trust calibration
• large language models