Optimizing Social Media Annotation of HPV Vaccine Skepticism and Misinformation Using Large Language Models: An Experimental Evaluation of In-Context Learning and Fine-Tuning Stance Detection Across Multiple Models

2024-11-22

arXiv.org

AI Summary

This study addresses the low annotation efficiency and high labor cost associated with stance detection in HPV vaccine–related tweets on social media. We systematically compare in-context learning (ICL) against supervised fine-tuning across mainstream large language models (LLMs), including GPT-4, Mistral, and Llama3. Through ablation studies, we find—novelly for this task—that ICL consistently outperforms fine-tuning. The optimal configuration employs six hierarchically sampled, high-quality exemplars paired with comprehensive contextual prompts, achieving an F1 score of 0.82—4.7 percentage points higher than the fine-tuning baseline. We further uncover differential sensitivity of LLMs to ICL prompt structure. Our results demonstrate strong cross-model generalizability of the ICL approach and establish a reusable, low-cost, high-accuracy framework for annotating stances toward vaccine-related misinformation.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This paper leverages large-language models (LLMs) to experimentally determine optimal strategies for scaling up social media content annotation for stance detection on HPV vaccine-related tweets. We examine both conventional fine-tuning and emergent in-context learning methods, systematically varying strategies of prompt engineering across widely used LLMs and their variants (e.g., GPT4, Mistral, and Llama3, etc.). Specifically, we varied prompt template design, shot sampling methods, and shot quantity to detect stance on HPV vaccination. Our findings reveal that 1) in general, in-context learning outperforms fine-tuning in stance detection for HPV vaccine social media content; 2) increasing shot quantity does not necessarily enhance performance across models; and 3) different LLMs and their variants present differing sensitivity to in-context learning conditions. We uncovered that the optimal in-context learning configuration for stance detection on HPV vaccine tweets involves six stratified shots paired with detailed contextual prompts. This study highlights the potential and provides an applicable approach for applying LLMs to research on social media stance and skepticism detection.

Problem

• Optimizing stance detection on HPV vaccine tweets using LLMs
• Comparing in-context learning and fine-tuning for misinformation detection
• Evaluating prompt engineering strategies across multiple LLM variants

Innovation

• Uses LLMs for HPV vaccine stance detection
• Compares in-context learning and fine-tuning
• Optimizes prompt design and shot quantity