Technology Category
Machine Learning: Ethics, Bias, and FairnessNatural Language Processing: Ethics — Bias, Fairness, Transparency & PrivacyHumans and AI: Crowd Sourcing and Human Computation
Wisdom from Diversity: Bias Mitigation Through Hybrid Human-LLM Crowds
🤖 AI Summary
Large language models (LLMs) often inherit societal biases from training data, exacerbating unfairness—particularly along gender and ethnic dimensions—when responding to sensitive prompts. To address this, we propose the first hybrid crowdsourcing framework that jointly optimizes human diversity and LLM accuracy for bias mitigation. Our method introduces three key innovations: (1) the first systematic demonstration that purely LLM-based crowdsourcing amplifies, rather than reduces, bias; (2) a locally weighted response aggregation mechanism that dynamically calibrates individual response biases; and (3) a human-AI collaborative, cross-subject response integration paradigm. Extensive experiments on multiple ethical evaluation benchmarks show that our approach significantly outperforms state-of-the-art baselines, achieving a new SOTA in bias reduction (average 32.7% decrease) while simultaneously improving the accuracy–fairness trade-off.
Technology Category
Application Category
📝 Abstract
Despite their performance, large language models (LLMs) can inadvertently perpetuate biases found in the data they are trained on. By analyzing LLM responses to bias-eliciting headlines, we find that these models often mirror human biases. To address this, we explore crowd-based strategies for mitigating bias through response aggregation. We first demonstrate that simply averaging responses from multiple LLMs, intended to leverage the"wisdom of the crowd", can exacerbate existing biases due to the limited diversity within LLM crowds. In contrast, we show that locally weighted aggregation methods more effectively leverage the wisdom of the LLM crowd, achieving both bias mitigation and improved accuracy. Finally, recognizing the complementary strengths of LLMs (accuracy) and humans (diversity), we demonstrate that hybrid crowds containing both significantly enhance performance and further reduce biases across ethnic and gender-related contexts.
Problem
Research questions and friction points this paper is trying to address.
Mitigating biases in LLMs through hybrid human-LLM crowds
Addressing bias exacerbation in LLM response aggregation
Enhancing accuracy and diversity via hybrid crowd strategies
Innovation
Methods, ideas, or system contributions that make the work stand out.
Hybrid human-LLM crowds reduce biases
Locally weighted aggregation mitigates bias
Combining human diversity with LLM accuracy
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
No related jobs found.
