Benchmarking Argumentative Behaviour of LLMs: A Study of Defences Against Character Attacks

2026-09-23

AI Summary

This study addresses the challenge that large language models (LLMs) face in strategically deploying ethical rebuttals against ad hominem attacks during political debates, a capability inherent to human interlocutors. By innovatively conceptualizing ad hominem attacks as pragmatic norms of political discourse rather than mere logical fallacies, this work constructs a conversational game-theoretic benchmark based on the ElecDeb60to16 corpus to systematically compare the defensive strategies of LLMs and human candidates. The findings reveal that safety fine-tuning significantly constrains the strategic action space of LLMs. Most models exhibit a rigid reliance on logical defenses while lacking the capacity for ethical counterattacks, demonstrating that current LLMs remain inadequate for navigating the complex discursive dynamics characteristic of authentic political interactions.

Abstract

Large Language Models (LLMs) are increasingly deployed as argumentative agents in persuasive dialogues, necessitating rigorous evaluation of their debating competence relative to human interlocutors. In this study, we focus on character attacks (ad hominem arguments), traditionally dismissed as fallacies, which play a pivotal role in political persuasive dialogues where ethos often rivals propositional content. Specifically, we investigate whether modern LLMs can replicate human competence to strategically use and respond to such attacks. We analyse a corpus of natural language political dialogues to identify defensive strategies human interlocutors naturally employ in ethos-centred debates and structure them into a dialogue game. Empirically, we benchmark LLM-generated dialogues against the ElecDeb60to16-fallacy corpus of U.S. presidential debates, contrasting human debaters' repertoire of defensive strategies with those of artificial agents. Results reveal a substantial difference: most LLMs rigidly prioritise logical defences, failing to exploit ethotic counterattacks as valid moves in political discourse. We argue that current safety fine-tuning constraints the strategic action space of these LLMs, making them unable to fully engage in naturalistic interactions within domains where character contestation is a normative expectation rather than a mere fallacy.

Problem

• Large Language Models
• Argumentative Agents
• Character Attacks
• Ad Hominem
• Persuasive Dialogues

Innovation

• Argumentative Agents
• Ad Hominem Defences
• Dialogue Game
• Safety Fine-tuning
• Benchmarking LLMs