Evaluating Communicative Belief Updates in Large Language Models via Implicature Recognition and Cancellation

2026-07-27

AI Summary

This study addresses a critical gap in large language models’ (LLMs) pragmatic competence—specifically, their ability to interpret conversational implicatures and update implicit beliefs when those implicatures are canceled. The authors present the first systematic evaluation of this capability, introducing a novel expert-annotated dataset of implicature–cancellation pairs. Combining linguistic theory, crowdsourced annotations, and controlled experiments, they demonstrate that model performance is heavily influenced by prior beliefs and varies significantly across implicature types and surface forms. To support future research, the work also releases a high-quality, human-annotated benchmark dataset along with human judgment baselines, establishing a foundational resource for advancing LLMs’ understanding of pragmatic inference and belief revision in dialogue.

Abstract

Human language is driven by unspoken beliefs and belief updates, making these critical to model for successful communication between large language models (LLMs) and their users. In this paper, we evaluate the ability of LLMs to recognize unspoken beliefs made through implicatures and to understand their updates through implicature cancellation: the pragmatic phenomenon whereby an utterance's implied meaning is weakened or negated. We create the first expert-annotated implicature cancellation dataset, [DatasetName], crowdsourced for human judgements of implicatures and their corresponding cancellations. We find that LLM belief update understanding lags behind that of humans, especially in more naturally-occurring scenarios. Additional control experiments suggest that successes in LLM belief updates may stem in part from a reliance on prior beliefs, and that failures in belief updates may depend on their type and on their form. Overall, our study suggests that current LLMs have not yet reached human-level understanding of unspoken beliefs and belief updates. Code and data are available at https://github.com/cesare-spinoso/ImplicatureX.

Problem

• implicature
• belief update
• large language models
• pragmatics
• implicature cancellation

Innovation

• implicature cancellation
• belief update
• pragmatic reasoning
• large language models
• human-annotated dataset