Long Live Fine-Tuning: Task-Specific Transformers Outperform Zero-Shot LLMs for Misinformation Response Classification on Reddit

2026-06-02

AI Summary

This study investigates whether large language models (LLMs) can effectively perform fine-grained classification of user responses to misinformation on Reddit—such as amplification or debunking—across three PolitiFact-verified misinformation topics. Comparing zero-shot LLMs (e.g., Llama, Claude, Gemini) against fine-tuned Transformer models (e.g., DistilBERT, RoBERTa), the findings reveal that model scale is not decisive: a fine-tuned RoBERTa achieves a macro F1 score of 0.62 at substantially lower computational cost, significantly outperforming the best zero-shot LLM (0.50). The results highlight the critical advantage of task-specific fine-tuning, particularly in identifying implicit “belief” categories, while also indicating that safety alignment mechanisms can impair LLMs’ ability to detect misinformation in sensitive content.

Abstract

As large language models (LLMs) become default tools for online information verification, an implicit assumption follows them: that scale and general capability are sufficient for nuanced classification of misinformation discourse. We test this assumption directly on 900 Reddit comments spanning three PolitiFact-verified misinformation claims (environment, health, immigration), labelled as belief (propagates the claim), fact-check (corrects it), or other. We compare nine models across three paradigms -- BART-MNLI, three Llama variants, three commercial frontier LLMs (Claude Haiku 4.5, Gemini Flash Lite 2.5, Claude Sonnet 4.6), and fine-tuned DistilBERT and RoBERTa -- under universal and topic-specific label schemas. The assumption does not hold. Fine-tuned RoBERTa reaches 0.62 macro-$F_1$ against a best zero-shot result of 0.50 (Claude Haiku 4.5), at a fraction of the per-query cost; the supervised advantage is concentrated on the belief class, the implicit, affective category every zero-shot model under-detects. Scaling does not help: Llama-3-8B matches Llama-3-70B, and Claude Sonnet 4.6 underperforms the smaller Haiku under generic labels, collapsing belief detection to 0.17 and refusing outright on a subset of comments flagged as sensitive. This is a safety-alignment artefact, not a capacity limit. Label schema and topic jointly shape zero-shot performance, with the same model varying by more than 0.13 macro-$F_1$ across topics under matched labels. In a verification context, where missing belief is the costlier error, task-specific fine-tuning remains the more reliable choice despite the proliferation of large generative models.

Problem

• misinformation classification
• zero-shot LLMs
• fine-tuning
• belief detection
• online discourse

Innovation

• fine-tuning
• misinformation classification
• zero-shot LLMs
• task-specific transformers
• belief detection