Clarifying Misconceptions in COVID-19 Vaccine Sentiment and Stance Analysis and Their Implications for Vaccine Hesitancy Mitigation: A Systematic Review

2025-03-23

AI Summary

This study addresses pervasive measurement bias, annotation inconsistency, and misinterpretation of results in COVID-19 vaccine sentiment and stance analysis. We systematically reviewed 67 NLP/ML studies (2020–2023) leveraging Twitter data. Introducing the first five-dimensional taxonomy—covering sampling strategy, study design, classification schema, annotation protocol, and result interpretation—we clarify fundamental distinctions and methodological flaws between sentiment analysis and stance detection in vaccine hesitancy assessment. Guided by the PROSPERO registration framework and an NLP methodology quality audit, we found that over 80% of studies suffer from measurement bias, severely limiting generalizability. Based on these findings, we propose a public health–oriented NLP reporting standard to enhance transparency, reproducibility, and validity. This work provides a methodological foundation for trustworthy vaccine hesitancy monitoring and evidence-informed interventions.

Technology Category

Natural Language Processing: Ethics — Bias, Fairness, Transparency & PrivacyMachine Learning: Ethics, Bias, and FairnessConstraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

Social Networks and Social Media: Fairness and bias in social network and social media analysisSearch and Retrieval-Augmented AI: Web evaluation methodologies and metricsWeb Mining and Content Analysis: Sentiment analysis and opinion mining

Abstract

Background Advances in machine learning (ML) models have increased the capability of researchers to detect vaccine hesitancy in social media using Natural Language Processing (NLP). A considerable volume of research has identified the persistence of COVID-19 vaccine hesitancy in discourse shared on various social media platforms. Methods Our objective in this study was to conduct a systematic review of research employing sentiment analysis or stance detection to study discourse towards COVID-19 vaccines and vaccination spread on Twitter (officially known as X since 2023). Following registration in the PROSPERO international registry of systematic reviews, we searched papers published from 1 January 2020 to 31 December 2023 that used supervised machine learning to assess COVID-19 vaccine hesitancy through stance detection or sentiment analysis on Twitter. We categorized the studies according to a taxonomy of five dimensions: tweet sample selection approach, self-reported study type, classification typology, annotation codebook definitions, and interpretation of results. We analyzed if studies using stance detection report different hesitancy trends than those using sentiment analysis by examining how COVID-19 vaccine hesitancy is measured, and whether efforts were made to avoid measurement bias. Results Our review found that measurement bias is widely prevalent in studies employing supervised machine learning to analyze sentiment and stance toward COVID-19 vaccines and vaccination. The reporting errors are sufficiently serious that they hinder the generalisability and interpretation of these studies to understanding whether individual opinions communicate reluctance to vaccinate against SARS-CoV-2. Conclusion Improving the reporting of NLP methods is crucial to addressing knowledge gaps in vaccine hesitancy discourse.

Problem

• Analyzing COVID-19 vaccine hesitancy via sentiment and stance on Twitter
• Identifying measurement bias in supervised ML studies on vaccine discourse
• Improving NLP methods to accurately assess vaccine reluctance trends

Innovation

• Used supervised machine learning for sentiment analysis
• Applied stance detection on Twitter data
• Systematically reviewed NLP methods for bias