Rethinking Ground Truth: A Case Study on Human Label Variation in MLLM Benchmarking

2026-03-20

AI Summary

Current benchmarks for multimodal large language models (MLLMs) commonly overlook systematic discrepancies in human annotator judgments—referred to as human label variation (HLV)—leading to an overestimation of model capabilities. This work proposes a novel evaluation protocol that explicitly models HLV within MLLM benchmarks for the first time, partitioning data into high- and low-agreement subsets based on non-aggregated human annotations. Using this framework, we conduct conditional evaluations of state-of-the-art models, including Gemma 3 and Qwen 2.5 VL, on a social media content classification task. Our findings reveal that while large models excel on high-agreement data, they often underperform medium-sized models on high-disagreement instances, demonstrating that model scale does not necessarily correlate with improved handling of subjective or ambiguous inputs. Consequently, reliance solely on consensus labels substantially inflates perceived model performance.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Human Label Variation (HLV), i.e. systematic differences among annotators' judgments, remains underexplored in benchmarks despite rapid progress in large language model (LLM) development. We address this gap by introducing an evaluation protocol for multimodal large language model (MLLM) benchmarking that explicitly accounts for two conditions: (1) human label agreement and (2) disagreement. We apply this protocol to two state-of-the-art MLLM families (Gemma 3, Qwen 2.5 VL) using non-aggregated human annotations from a social media content classification dataset. Across tasks, we find that larger models tend to perform best on high-agreement subsets, yet often underperform medium-sized models when human disagreement is high, indicating that parameter count alone does not determine sensitivity to ambiguity and subjectivity. These results show that benchmarks based solely on consensus labels can overstate model capabilities in such domains and that incorporating human label variation yields more realistic and robust assessments of MLLMs in content moderation pipelines.

Problem

• Human Label Variation
• MLLM Benchmarking
• Label Agreement
• Subjectivity
• Content Moderation

Innovation

• Human Label Variation
• MLLM Benchmarking
• Evaluation Protocol
• Label Disagreement
• Content Moderation