The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Prediction

2026-06-01

AI Summary

This study addresses the lack of systematic analysis regarding uncertainty in large language model (LLM)–generated annotations and their divergence from human judgments. The authors propose the Ghost Annotator framework, which integrates conformal prediction with collaboratively filtered annotator embeddings to model patterns of agreement and disagreement between LLMs and human annotators. They introduce a novel metric, Ghost Prediction, to quantify instances where model predictions deviate from all human labels. Evaluations across four content moderation datasets and four LLM families reveal that model uncertainty generally increases with human annotator disagreement; however, larger models often exhibit overconfidence on inputs unsupported by any human annotator and display persistent demographic-based structural biases.

Abstract

Current research primarily focuses on model performance, while comparatively less attention has been devoted to uncertainty estimation, particularly in settings where LLMs are increasingly used to generate annotated data. We introduce a framework combining conformal prediction with Collaborative Filtering-style annotators' representation to model LLM behavior in relation to human annotators and to analyze patterns of agreement and disagreement. Using Non-Conformity Scores, we introduce the Ghost Prediction metric and the Ghost Annotator representation to quantify cases in which model predictions diverge from all available human annotations. We compute cosine similarity measures to explore differences in model behavior across sociodemographic axes. We evaluated four LLMs of different size and families across four content moderation datasets. Our finding shows that while we find that all models uncertainty increases with annotator disagreement, larger models tend to be more confident in the classification of texts that are not aligned with any human annotation. Finally, the Ghost Annotator framework reveals a consistent and robust pattern of demographic misalignment, suggesting a structural bias likely rooted in pretraining corpora.

Problem

• uncertainty estimation
• human label variation
• content moderation
• demographic bias
• annotation disagreement

Innovation

• Conformal Prediction
• Ghost Annotator
• Label Variation
• Content Moderation
• Demographic Bias