How Large Language Models play humans in online conversations: a simulated study of the 2016 US politics on Reddit

2025-06-23

AI Summary

This study systematically evaluates the capabilities and risks of large language models (LLMs) in generating partisan comments within realistic political polarization contexts—specifically, Reddit discussions during the 2016 U.S. presidential election. Using GPT-4 with role-playing prompt engineering, we generate synthetic partisan comments and assess them through multi-dimensional analysis: political stance classification, sentiment analysis, LIWC linguistic feature profiling, and BERT-based semantic embedding clustering. Our key findings are threefold: (1) LLM-generated comments exhibit high verisimilitude and ideological consistency but remain linearly separable from human-authored comments in semantic space; (2) they disproportionately reinforce consensus rather than stimulate dissent; and (3) human annotators distinguish synthetic from authentic comments at only 51% accuracy—significantly below the chance-level threshold. These results reveal a concrete, stealthy risk of LLM-mediated manipulation in online political discourse and provide critical empirical evidence for AI governance frameworks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Humans and AI: Voting

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Large Language Models (LLMs) have recently emerged as powerful tools for natural language generation, with applications spanning from content creation to social simulations. Their ability to mimic human interactions raises both opportunities and concerns, particularly in the context of politically relevant online discussions. In this study, we evaluate the performance of LLMs in replicating user-generated content within a real-world, divisive scenario: Reddit conversations during the 2016 US Presidential election. In particular, we conduct three different experiments, asking GPT-4 to generate comments by impersonating either real or artificial partisan users. We analyze the generated comments in terms of political alignment, sentiment, and linguistic features, comparing them against real user contributions and benchmarking against a null model. We find that GPT-4 is able to produce realistic comments, both in favor of or against the candidate supported by the community, yet tending to create consensus more easily than dissent. In addition we show that real and artificial comments are well separated in a semantically embedded space, although they are indistinguishable by manual inspection. Our findings provide insights on the potential use of LLMs to sneak into online discussions, influence political debate and shape political narratives, bearing broader implications of AI-driven discourse manipulation.

Problem

• Evaluates LLMs' ability to mimic human political discussions on Reddit
• Assesses GPT-4's realism in generating partisan comments during 2016 US election
• Examines AI's potential to influence political narratives in online debates

Innovation

• GPT-4 generates partisan Reddit comments
• Analyzes political alignment and sentiment
• Compares real and artificial comment semantics