Contextualized Prompting For Stance Detection On Social Media

2026-06-04

AI Summary

This study addresses the challenge of stance detection on social media, where short, noisy texts and strong contextual dependencies hinder performance—particularly in zero-shot settings with large language models (LLMs). The authors present the first systematic evaluation of three types of contextual features: real-world (e.g., user bios), derived (e.g., party affiliation), and LLM-generated (e.g., target descriptions). By designing multi-context prompt templates and evaluating across four benchmark datasets—including a newly introduced high-quality German Twitter dataset—they demonstrate that LLM-generated target descriptions consistently and significantly improve accuracy. In contrast, other forms of user metadata offer limited benefits and can even degrade performance, revealing that LLMs struggle to discern task-relevant from irrelevant contextual information in zero-shot stance detection.

Abstract

Stance detection on social media is challenging due to short, noisy, and context-dependent language. While large language models (LLMs) show zero-shot generalization, they are typically prompted without contextual information, which limits their ability to interpret ambiguous posts. In this work, we systematically investigate the impact of incorporating real-world (e.g., user biographies), derived (e.g., political party), and LLM-generated (e.g., target descriptions) contextual features into zero-shot prompting for stance detection on Twitter. Our evaluation spans four benchmark datasets, including a new high-quality German Twitter stance dataset. Across multiple LLMs, we find that integrating contextual information improves performance, but only under specific conditions. LLM-generated target descriptions consistently enhance accuracy, while other user metadata has mixed or even detrimental effects. Notably, we show that the inclusion of other tweets by the same user, often beneficial in supervised learning, can impair performance due to input noise. Our qualitative analysis reveals that LLMs struggle to distinguish task-specific useful information from irrelevant context. Our findings highlight both the promise and challenges of prompting with context information in noisy real-world settings. We publish code and data at this \href{https://github.com/tilmanbeck/stance-context-twitter}{page}.

Problem

• stance detection
• social media
• contextual information
• large language models
• zero-shot prompting

Innovation

• contextualized prompting
• stance detection
• zero-shot learning
• large language models
• social media