On the Interplay between Human Label Variation and Model Fairness

2025-10-13

AI Summary

This study presents the first systematic investigation into how human label variation (HLV) affects the fairness of machine learning models. Addressing the limitation of majority-voting labels—which obscure annotator disagreement—the work employs multiple HLV modeling strategies, including probabilistic labels, distributional labels, and multi-annotator ensembles, to train state-of-the-art classifiers on standard benchmark datasets. Critically, no explicit debiasing mechanisms are introduced during training. The models are rigorously evaluated for both predictive accuracy and fairness across group-level (e.g., demographic parity, equalized odds) and individual-level (e.g., counterfactual fairness) metrics. Experimental results demonstrate that HLV-based models consistently achieve superior fairness performance compared to majority-voting baselines, while maintaining comparable—or even higher—accuracy. The key contribution is the empirical revelation that naturally occurring label diversity inherently possesses debiasing potential, thereby establishing a novel, intervention-free paradigm for fair machine learning.

Technology Category

• Machine Learning: Ethics, Bias, and Fairness
• Humans and AI: Human-in-the-loop Machine Learning
• Computer Vision: Bias, Fairness & Privacy

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

The impact of human label variation (HLV) on model fairness is an unexplored topic. This paper examines the interplay by comparing training on majority-vote labels with a range of HLV methods. Our experiments show that without explicit debiasing, HLV training methods have a positive impact on fairness.

Problem

• Explores how human label variation affects model fairness outcomes
• Compares majority-vote labels with HLV training methodologies
• Demonstrates HLV methods improve fairness without explicit debiasing

Innovation

• Compares majority-vote labels with HLV methods
• Examines HLV training impact on model fairness
• HLV methods improve fairness without debiasing