A Multi-Level Benchmark for Causal Language Understanding in Social Media Discourse

2025-09-20

AI Summary

Existing causal inference research primarily targets explicit causal relations in formal, structured text, struggling with informal and implicit causal expressions prevalent in social media. Method: We introduce CausalTalk—the first multi-level causal understanding benchmark built on five years of COVID-19–related Reddit posts—encompassing four tasks: causal classification, explicit/implicit distinction, span extraction, and key-point generation. Leveraging a hybrid annotation strategy combining domain experts and GPT-4o, we ensure both high quality and scalability, yielding a dataset of 10,120 finely annotated instances. Contribution/Results: CausalTalk is the first benchmark to jointly model and evaluate both explicit and implicit causal structures in informal discourse, addressing the critical lack of standardized evaluation frameworks for causal reasoning in social text. It provides a reproducible, comparable testbed for both discriminative and generative models, enabling rigorous assessment of causal understanding in real-world, unstructured communication.

Technology Category

• Reasoning under Uncertainty: Causality
• Machine Learning: Causal Learning
• Knowledge Representation and Reasoning: Action, Change, and Causality

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Understanding causal language in informal discourse is a core yet underexplored challenge in NLP. Existing datasets largely focus on explicit causality in structured text, providing limited support for detecting implicit causal expressions, particularly those found in informal, user-generated social media posts. We introduce CausalTalk, a multi-level dataset of five years of Reddit posts (2020-2024) discussing public health related to the COVID-19 pandemic, among which 10120 posts are annotated across four causal tasks: (1) binary causal classification, (2) explicit vs. implicit causality, (3) cause-effect span extraction, and (4) causal gist generation. Annotations comprise both gold-standard labels created by domain experts and silver-standard labels generated by GPT-4o and verified by human annotators. CausalTalk bridges fine-grained causal detection and gist-based reasoning over informal text. It enables benchmarking across both discriminative and generative models, and provides a rich resource for studying causal reasoning in social media contexts.

Problem

• Detecting implicit causal expressions in informal social media discourse
• Addressing limited datasets for causal language understanding in user-generated content
• Benchmarking causal reasoning across discriminative and generative models

Innovation

• Multi-level dataset for implicit causality detection
• Combines expert annotations with GPT-4o generated labels
• Supports both discriminative and generative model benchmarking