Implicit Bias in LLMs: A Survey

2025-03-04

AI Summary

This paper addresses the challenge of detecting implicit biases in large language models (LLMs)—subtle, unconscious biases that evade explicit detection yet systematically influence model outputs. To address the lack of a systematic framework in prior work, we formally adapt psychological theories of implicit cognition—particularly the Implicit Association Test (IAT)—to the LLM domain. We propose a three-tier taxonomy encompassing detection methodologies, evaluation metrics, and benchmark datasets. Specifically, we introduce three detection paradigms—word association, task-oriented generation, and decision-making—and two metric categories—single-value and comparative-value measures. By unifying IAT-inspired probes, word embedding analyses, masked/complete-sentence benchmarks, and task-driven generation evaluations, we establish the first comprehensive, unified taxonomy for assessing implicit bias in LLMs. This work provides a theoretical foundation, methodological guidance, and practical tools for advancing trustworthy AI research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Due to the implement of guardrails by developers, Large language models (LLMs) have demonstrated exceptional performance in explicit bias tests. However, bias in LLMs may occur not only explicitly, but also implicitly, much like humans who consciously strive for impartiality yet still harbor implicit bias. The unconscious and automatic nature of implicit bias makes it particularly challenging to study. This paper provides a comprehensive review of the existing literature on implicit bias in LLMs. We begin by introducing key concepts, theories and methods related to implicit bias in psychology, extending them from humans to LLMs. Drawing on the Implicit Association Test (IAT) and other psychological frameworks, we categorize detection methods into three primary approaches: word association, task-oriented text generation and decision-making. We divide our taxonomy of evaluation metrics for implicit bias into two categories: single-value-based metrics and comparison-value-based metrics. We classify datasets into two types: sentences with masked tokens and complete sentences, incorporating datasets from various domains to reflect the broad application of LLMs. Although research on mitigating implicit bias in LLMs is still limited, we summarize existing efforts and offer insights on future challenges. We aim for this work to serve as a clear guide for researchers and inspire innovative ideas to advance exploration in this task.

Problem

• Study implicit bias in large language models (LLMs).
• Categorize detection methods using psychological frameworks.
• Summarize efforts to mitigate implicit bias in LLMs.

Innovation

• Uses Implicit Association Test for bias detection
• Classifies bias metrics into single and comparison values
• Incorporates diverse datasets for comprehensive evaluation