Decoding AI Judgment: How LLMs Assess News Credibility and Bias

2025-02-06

AI Summary

This study investigates the internal mechanisms by which large language models (LLMs) assess news credibility and political bias. To this end, we conduct a multi-model benchmark evaluation using Gemini 1.5 Flash, GPT-4o mini, and LLaMA 3.1, integrating lexical frequency and contextual attribution analysis, rank-distribution statistics, retrieval-augmented reasoning (RAG), and cross-model consensus validation to systematically decode linguistic features driving model decisions. Results reveal that LLMs rely predominantly on superficial linguistic cues—not factual verification—for credibility judgments; political bias classification achieves up to 78% accuracy across models but exhibits pronounced systematic biases. Our key contribution is a novel dynamic evaluation framework supporting external retrieval integration, cross-model querying, and response-adaptive optimization—enabling, for the first time, an interpretable, feature-level deconstruction of how LLMs evaluate news credibility.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Language Models (LLMs) are increasingly used to assess news credibility, yet little is known about how they make these judgments. While prior research has examined political bias in LLM outputs or their potential for automated fact-checking, their internal evaluation processes remain largely unexamined. Understanding how LLMs assess credibility provides insights into AI behavior and how credibility is structured and applied in large-scale language models. This study benchmarks the reliability and political classifications of state-of-the-art LLMs - Gemini 1.5 Flash (Google), GPT-4o mini (OpenAI), and LLaMA 3.1 (Meta) - against structured, expert-driven rating systems such as NewsGuard and Media Bias Fact Check. Beyond assessing classification performance, we analyze the linguistic markers that shape LLM decisions, identifying which words and concepts drive their evaluations. We uncover patterns in how LLMs associate credibility with specific linguistic features by examining keyword frequency, contextual determinants, and rank distributions. Beyond static classification, we introduce a framework in which LLMs refine their credibility assessments by retrieving external information, querying other models, and adapting their responses. This allows us to investigate whether their assessments reflect structured reasoning or rely primarily on prior learned associations.

Problem

• LLMs' internal credibility assessment processes
• Linguistic markers influencing LLM decisions
• Refinement of credibility assessments through external information

Innovation

• LLMs benchmark against expert systems
• Analyze linguistic markers for credibility
• Framework for dynamic credibility assessment refinement