Large Means Left: Political Bias in Large Language Models Increases with Their Number of Parameters

2025-05-07

AI Summary

Large language models (LLMs) exhibit systematic political bias, yet its nature, drivers, and cross-model patterns remain poorly understood. Method: Focusing on the German federal election context, we quantitatively assess alignment between 12 mainstream LLMs and political parties using the official Wahl-O-Mat policy-position scoring framework. Contribution/Results: We identify three key findings: (1) a statistically significant positive correlation between model parameter count and leftward ideological bias—constituting the first empirical validation of scale-induced political skew; (2) prompt language, training-region origin, and release date as critical moderating factors, with multilingual prompting enabling dynamic modulation of bias intensity; and (3) consistent, systematic deviation of all models from actual electoral outcomes, where bias magnitude and direction closely track parties’ positions along the ideological spectrum. Our work establishes a reproducible, cross-model attribution framework for evaluating LLM political neutrality, grounded in real-world electoral data and standardized policy metrics.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

With the increasing prevalence of artificial intelligence, careful evaluation of inherent biases needs to be conducted to form the basis for alleviating the effects these predispositions can have on users. Large language models (LLMs) are predominantly used by many as a primary source of information for various topics. LLMs frequently make factual errors, fabricate data (hallucinations), or present biases, exposing users to misinformation and influencing opinions. Educating users on their risks is key to responsible use, as bias, unlike hallucinations, cannot be caught through data verification. We quantify the political bias of popular LLMs in the context of the recent vote of the German Bundestag using the score produced by the Wahl-O-Mat. This metric measures the alignment between an individual's political views and the positions of German political parties. We compare the models' alignment scores to identify factors influencing their political preferences. Doing so, we discover a bias toward left-leaning parties, most dominant in larger LLMs. Also, we find that the language we use to communicate with the models affects their political views. Additionally, we analyze the influence of a model's origin and release date and compare the results to the outcome of the recent vote of the Bundestag. Our results imply that LLMs are prone to exhibiting political bias. Large corporations with the necessary means to develop LLMs, thus, knowingly or unknowingly, have a responsibility to contain these biases, as they can influence each voter's decision-making process and inform public opinion in general and at scale.

Problem

• Quantifying political bias in large language models (LLMs)
• Investigating left-leaning bias correlation with model size
• Analyzing factors influencing LLM political preferences and misinformation risks

Innovation

• Quantify political bias using Wahl-O-Mat scores
• Analyze bias factors: size, origin, release date
• Evaluate communication language impact on bias