Are LLMs ready for HardChoices?

2026-07-13

AI Summary

This study investigates whether large language models (LLMs) exhibit robust and consistent stances on complex social issues characterized by intra-ideological disagreement. To this end, the authors introduce HardChoices, the first dataset specifically designed to capture such internal ideological tensions, and employ a systematic evaluation framework combining LLM-based question answering, stance classification, and consistency detection. The findings reveal that LLMs rarely adopt neutral positions and frequently display logical inconsistencies; however, when they do take explicit stances, their responses demonstrate high levels of convergence. This work uncovers a paradox in LLM behavior—simultaneous fragility and consensus—in expressing positions on deep-seated social issues, offering new insights into the models’ capacities and limitations in socio-political reasoning.

Abstract

A lot of research attention has been devoted to checking whether large language models (LLMs) are politically biased. This work has largely focused on high-level ideological dimensions, such as left--right or progressive--conservative, and it has been shown that while LLMs are predominantly left and progressive leaning, largely mimicking the biases in the training data, they can be to some extent steered to change their preferences in post-training. In this short note, we check if LLMs have robust stances with regard to major substantive societal issues, on which members of the same ideological camp are often in disagreement, summarised in a novel dataset \textsc{HardChoices}. We show that, faced with this line of questioning, LLMs, both large and small, surprisingly rarely declare neutrality, are often incoherent, and demonstrate a remarkable degree of agreement on issues where they do take stances.

Problem

• large language models
• political bias
• ideological coherence
• societal issues
• HardChoices

Innovation

• HardChoices
• large language models
• ideological bias
• stance consistency
• societal issues