Paired Completion: Flexible Quantification of Issue-framing at Scale with LLMs

2024-08-19

arXiv.org

AI Summary

This work addresses key challenges in automated framing analysis—particularly for polarized topics (e.g., climate science vs. denialism)—including poor generalizability across domains, heavy reliance on explicit lexical cues, and degraded performance under low-resource conditions. We propose “paired completion,” a novel paradigm leveraging next-token log-probabilities from large language models (LLMs). By integrating prompt engineering, contextual embeddings, and traditional NLP features, our method decouples framing detection from surface-level lexical differences, enabling few-shot (as few as 3–5 examples per stance), low-bias, and highly scalable frame quantification. Evaluated across 192 experimental configurations, it consistently outperforms state-of-the-art prompting strategies and supervised learning baselines. It achieves high accuracy and robustness on both synthetic and real-world corpora. We further conduct systematic analyses of bias sources and computational cost, establishing the first production-ready, efficient, trustworthy, and reproducible framework analysis methodology.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Preferences

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Detecting and quantifying issue framing in textual discourse - the perspective one takes to a given topic (e.g. climate science vs. denialism, misogyny vs. gender equality) - is highly valuable to a range of end-users from social and political scientists to program evaluators and policy analysts. However, conceptual framing is notoriously challenging for automated natural language processing (NLP) methods since the words and phrases used by either `side' of an issue are often held in common, with only subtle stylistic flourishes separating their use. Here we develop and rigorously evaluate new detection methods for issue framing and narrative analysis within large text datasets. By introducing a novel application of next-token log probabilities derived from generative large language models (LLMs) we show that issue framing can be reliably and efficiently detected in large corpora with only a few examples of either perspective on a given issue, a method we call `paired completion'. Through 192 independent experiments over three novel, synthetic datasets, we evaluate paired completion against prompt-based LLM methods and labelled methods using traditional NLP and recent LLM contextual embeddings. We additionally conduct a cost-based analysis to mark out the feasible set of performant methods at production-level scales, and a model bias analysis. Together, our work demonstrates a feasible path to scalable, accurate and low-bias issue-framing in large corpora.

Problem

• Detect contrasting issue framing in text
• Overcome subtle linguistic differences in automation
• Provide scalable low-resource framing analysis solution

Innovation

• LLM next-token log probabilities for frame detection
• Cost-efficient alternative to prompt-based methods
• Scalable solution for low-resource settings