Working with Large Language Models to Enhance Messaging Effectiveness for Vaccine Confidence

2025-04-14

AI Summary

This study addresses the practical challenge of leveraging large language models (e.g., ChatGPT) to enhance vaccine confidence communication in resource-constrained primary public health settings. Method: We conducted the first empirical investigation using a double-blind, matched-pair experiment comparing AI-rewritten vaccine messages against original ones across persuasiveness, credibility, and acceptability. We systematically examined boundary conditions—including AI source disclosure and message ordering—employing automated message rewriting, mixed-method evaluation (quantitative scoring + qualitative feedback), and rigorous bias-control design. Contribution/Results: AI-enhanced messages significantly increased average persuasiveness, especially when content was moderately extended. No systematic public resistance to AI-generated health information was observed; participants’ awareness of AI authorship showed no significant association with their evaluations. The study provides a reproducible methodological framework and critical empirical evidence for AI-augmented health communication at the grassroots level.

Technology Category

• Humans and AI: Voting
• Natural Language Processing: Safety and Robustness
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search

Abstract

Vaccine hesitancy and misinformation are significant barriers to achieving widespread vaccination coverage. Smaller public health departments may lack the expertise or resources to craft effective vaccine messaging. This paper explores the potential of ChatGPT-augmented messaging to promote confidence in vaccination uptake. We conducted a survey in which participants chose between pairs of vaccination messages and assessed which was more persuasive and to what extent. In each pair, one message was the original, and the other was augmented by ChatGPT. At the end of the survey, participants were informed that half of the messages had been generated by ChatGPT. They were then asked to provide both quantitative and qualitative responses regarding how knowledge of a message's ChatGPT origin affected their impressions. Overall, ChatGPT-augmented messages were rated slightly higher than the original messages. These messages generally scored better when they were longer. Respondents did not express major concerns about ChatGPT-generated content, nor was there a significant relationship between participants' views on ChatGPT and their message ratings. Notably, there was a correlation between whether a message appeared first or second in a pair and its score. These results point to the potential of ChatGPT to enhance vaccine messaging, suggesting a promising direction for future research on human-AI collaboration in public health communication.

Problem

• Addressing vaccine hesitancy and misinformation barriers
• Enhancing vaccine messaging with ChatGPT for public health
• Evaluating AI-generated content impact on message persuasiveness

Innovation

• ChatGPT-augmented messaging for vaccine confidence
• Survey comparing original and ChatGPT-enhanced messages
• Human-AI collaboration in public health communication