Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation

2026-09-22

AI Summary

This study addresses the instability of large language model (LLM) annotations under varying task designs, revealing substantial “instrument uncertainty” that confidence scores fail to mitigate. By evaluating seven LLMs across twelve task designs on a sample of 3,000 tweets, this work systematically quantifies the impact of prompt engineering on annotation outcomes using Fleiss’ and Cohen’s Kappa coefficients. The findings demonstrate that design variations inflate the variance of prevalence estimates by 76- to 110-fold, far exceeding typical inter-annotator disagreement among humans. Furthermore, this research establishes that comparing multiple task designs is the only effective approach for measuring such uncertainty, exposing critical limitations in existing evaluation methodologies and highlighting systematic biases introduced by both model selection and prompt design.

Abstract

Large language models (LLMs) can give reliable labels under one setup yet change those labels when researchers make other reasonable design choices. We tested seven LLMs, 12 task designs, three independent runs, and 3,000 tweets labeled for offensive language and hate speech. Repeating the same model and task design produced high agreement (median Fleiss' $κ= 0.91$). Agreement fell when we changed the task design for the same tweets (median Cohen's $κ= 0.76$). Task design and model choice increased the variance of estimated prevalence by factors of 76.7 for offensive language and 110.6 for hate speech compared with sampling variance alone. Variation across LLM task designs reached 560-572 basis points, compared with 270-331 basis points across five human instrument versions. Confidence scores did not solve this problem. They tracked repeated model outputs more closely than agreement with human labels, and grouping six tweets in one prompt lowered mean offensive-language confidence by 660 basis points. We call the variation caused by task design and model choice instrument uncertainty. Researchers can measure it only by comparing reasonable task designs. Repeating one setup or relying on confidence scores cannot replace that test.

Problem

• Large Language Models
• Instrument Uncertainty
• Task Design Sensitivity
• Annotation Reliability
• Label Variance

Innovation

• Instrument Uncertainty
• LLM Annotation
• Task Design Sensitivity
• Offensive Language Detection
• Confidence Scores