Poli-Bias: Understanding and Measuring Large Language Model Biases in International Political Conflicts

2026-08-06

AI Summary

This study addresses the challenge of quantifying political bias in large language models (LLMs) when responding to international political conflicts, where biases often arise from differing national identities and remain difficult to measure. To this end, the paper introduces Poli-Bias, a novel framework that pioneers the use of counterfactual methods for assessing political bias. By systematically swapping national identities in prompts, the approach constructs paired inputs representing equivalent conflict scenarios and evaluates model responses across five interpretable dimensions. This enables fine-grained auditing of both political neutrality and sycophantic tendencies, overcoming the limitations of single-score evaluations. Experiments across 13 mainstream LLMs demonstrate that both the nationality embedded in the prompt and the user’s perceived affiliation significantly influence how models describe, evaluate, and justify conflict-related behaviors, thereby validating Poli-Bias as an effective and generalizable tool for uncovering political bias.

Abstract

Measuring political bias in large language models (LLMs) remains challenging as it can manifest through subtle differences in framing, argumentation, and legal reasoning that are difficult to capture with a single metric. In this work, we introduce Poli-Bias, a counterfactual framework for measuring whether LLMs treat legally equivalent conflict scenarios differently depending on the countries involved. Poli-Bias compares responses to paired prompts in which country identities are systematically swapped across diverse geopolitical relationships, legal violations, and reasoning tasks. Rather than reducing bias to a single judgment, our framework decomposes response disparities into five interpretable dimensions, revealing how and where unequal treatment manifests. Across 13 contemporary LLMs spanning diverse model families and sizes, we find that country identities and user affiliations can systematically affect how equivalent actions are described, evaluated, and defended under international law. Our results thus establish Poli-Bias as a fine-grained framework for auditing political even-handedness and sycophancy in LLMs.

Problem

• political bias
• large language models
• international conflicts
• counterfactual evaluation
• model fairness

Innovation

• counterfactual framework
• political bias
• large language models
• international law
• response disparity