How Persuasive Could LLMs Be? A First Study Combining Linguistic-Rhetorical Analysis and User Experiments

2025-08-13

AI Summary

This study investigates the rhetorical features and persuasive efficacy of ChatGPT-generated argumentative texts on ethically sensitive topics—such as gene editing and algorithmic bias—among human readers. Method: Employing an innovative mixed-methods approach, it integrates qualitative rhetorical analysis with a pretest–posttest controlled experiment, supplemented by surveys and close textual reading to systematically assess large language models’ mechanisms of persuasion in ethical deliberation. Contribution/Results: Findings reveal that while ChatGPT produces structurally coherent and logically consistent arguments, its persuasive impact is highly topic-dependent. Notably, on high-stakes ethical issues, its outputs fail to alleviate user concerns and instead intensify moral apprehension. The study thus uncovers intrinsic limitations of AI-generated discourse in value-laden negotiation, offering empirical evidence and a methodological framework for advancing AI ethics communication and designing human–AI deliberative systems.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Philosophy and Ethics of AI: Artificial General Intelligence
• Humans and AI: Learning Human Values and Preferences

Application Category

• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

This study examines the rhetorical and linguistic features of argumentative texts generated by ChatGPT on ethically nuanced topics and investigates their persuasive impact on human readers.Through a user study involving 62 participants and pre-post interaction surveys, the paper analyzes how exposure to AI-generated arguments affects opinion change and user perception. A linguistic and rhetorical analysis of the generated texts reveals a consistent argumentative macrostructure, reliance on formulaic expressions, and limited stylistic richness. While ChatGPT demonstrates proficiency in constructing coherent argumentative texts, its persuasive efficacy appears constrained, particularly on topics involving ethical issues.The study finds that while participants often acknowledge the benefits highlighted by ChatGPT, ethical concerns tend to persist or even intensify post-interaction. The results also demonstrate a variation depending on the topic. These findings highlight new insights on AI-generated persuasion in ethically sensitive domains and are a basis for future research.

Problem

• Examining persuasive impact of ChatGPT-generated ethical arguments on humans
• Analyzing linguistic-rhetorical features and user opinion change post-interaction
• Assessing limitations in AI persuasion efficacy on ethically nuanced topics

Innovation

• Combines linguistic-rhetorical analysis with user experiments
• Analyzes ChatGPT's argumentative macrostructure and style
• Measures persuasive impact via pre-post interaction surveys