Unequal Verdicts: Investigating Gender Bias in LLM-Based Fake News Detection

2026-08-04

AI Summary

This study addresses the underexplored issue of gender bias in large language models (LLMs) for fake news detection, which may compromise judgment fairness and reliability depending on the speaker’s gender. The authors present the first systematic investigation of this problem by constructing an augmented version of the LIAR dataset that includes neutral, male, and female variants of occupational titles. Through multiple prompt formulations, fairness metrics, and statistical significance tests, they evaluate the gender sensitivity of six prominent LLMs. Results reveal inconsistent predictions across all models (9.79%–35.13%), with five exhibiting significant directional bias—particularly a tendency to classify statements attributed to male speakers as more likely false. The work introduces the first benchmark dataset with gendered variants, offering a new resource for fairness evaluation in misinformation detection.

Abstract

Large Language Models (LLMs) are increasingly used for automated fact-checking, yet their susceptibility to gender bias in this context remains underexplored. This study presents the first systematic investigation of gender bias in LLM-based fake news detection using real-world data. We augment the LIAR benchmark with three gender variants of speaker job titles (Neutral, Male, Female) for each statement to test whether veracity judgments vary solely based on gender presentation. Six state-of-the-art LLMs are evaluated across multiple bias and fairness metrics. All models exhibit gender sensitivity: 9.79%-35.13% of statements receive inconsistent labels across the three variants, with Male-Female comparisons showing 6.5%-23.6% flip rates. Two primary bias manifestations are identified: instability (inconsistent judgments) and directionality (systematic favoritism). Five models show statistically significant directional effects, with the strongest effects displaying male-skeptic patterns. These findings demonstrate that gender bias undermines both reliability and fairness in LLM-based fake news detection, highlighting the need for bias-aware evaluation and mitigation strategies. The augmented dataset is publicly released to support future research.

Problem

• gender bias
• fake news detection
• large language models
• fairness
• fact-checking

Innovation

• gender bias
• fake news detection
• large language models
• fairness evaluation
• bias mitigation