Large Language Models are Perplexed by some Political Parties

2026-06-04

AI Summary

This study addresses systematic bias in large language models (LLMs) when processing texts reflecting diverse political ideologies, revealing consistently higher perplexity—indicative of greater processing difficulty—on far-right and nationalist party texts. For the first time, the authors evaluate fairness across ten prominent LLMs, including instruction-tuned variants, using three multilingual datasets covering 37 languages and political parties from multiple countries. Their analysis demonstrates that this bias predominantly originates during pretraining and is largely unmitigated by subsequent instruction tuning. Furthermore, the work establishes a quantitative link between perplexity and fairness in political text translation, offering a measurable foundation for understanding the implicit political inclinations embedded within LLMs.

Abstract

Large Language Models (LLMs) are increasingly used, including in political applications, but their political fairness has been little studied. We assess it using perplexity, posing that a fair model should give equal probability to all political groups. However, we find, across ten LLMs and three datasets covering 37 languages, that LLMs are more perplexed by the texts of far right and nationalist parties than of social-democratic parties. We find this to be consistent with previous work on translation fairness, to the point that perplexity correlates with downstream translation metrics. Our method is applicable to both base LLMs as well as their instruction-tuned counterpart, and we find that both are highly correlated, suggesting that the political fairness of LLMs stems from their pretraining, and is hardly affected by instruction-tuning.

Problem

• political fairness
• large language models
• perplexity
• political parties
• bias

Innovation

• political fairness
• perplexity
• large language models
• instruction tuning
• translation bias