Can AI-Generated Persuasion Be Detected? Persuaficial Benchmark and AI vs. Human Linguistic Differences

2026-01-08

arXiv.org

AI Summary

This study addresses the growing risk of misuse associated with AI-generated persuasive texts, which can be particularly difficult to detect due to their subtle and covert nature. To systematically investigate this challenge, the authors introduce Persuaficial, the first benchmark dataset specifically designed for multilingual persuasive text detection. Leveraging controlled generation techniques, multilingual NLP methods, and linguistic feature analysis, the work evaluates the detectability gap between human- and large language model–generated persuasive content. The findings reveal that while overtly AI-generated persuasive texts are relatively easy to identify, detection performance drops significantly when models employ implicit persuasion strategies. Furthermore, the study uncovers nuanced linguistic differences in how humans and AI deploy subtle persuasive tactics, offering new insights and a foundational resource for developing more interpretable and effective AI-generated text detection systems.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Humans and AI: AI for Accessibility
• Application Domains: Humanities & Computational Social Science

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Large Language Models (LLMs) can generate highly persuasive text, raising concerns about their misuse for propaganda, manipulation, and other harmful purposes. This leads us to our central question: Is LLM-generated persuasion more difficult to automatically detect than human-written persuasion? To address this, we categorize controllable generation approaches for producing persuasive content with LLMs and introduce Persuaficial, a high-quality multilingual benchmark covering six languages: English, German, Polish, Italian, French and Russian. Using this benchmark, we conduct extensive empirical evaluations comparing human-authored and LLM-generated persuasive texts. We find that although overtly persuasive LLM-generated texts can be easier to detect than human-written ones, subtle LLM-generated persuasion consistently degrades automatic detection performance. Beyond detection performance, we provide the first comprehensive linguistic analysis contrasting human and LLM-generated persuasive texts, offering insights that may guide the development of more interpretable and robust detection tools.

Problem

• AI-generated persuasion
• detection
• large language models
• human vs. AI text
• persuasive content

Innovation

• Persuaficial
• LLM-generated persuasion
• multilingual benchmark
• automatic detection
• linguistic analysis