Elucidating Mechanisms of Demographic Bias in LLMs for Healthcare

2025-02-18

AI Summary

This study investigates implicit sociodemographic encoding—particularly along gender and race dimensions—in large language models (LLMs) deployed in clinical settings, and its causal role in generating clinical bias. We propose the first mechanistic interpretability framework tailored to medical LLM bias analysis, integrating activation patching, intermediate-layer neuron probing, causal mediation analysis, and distributionally robust evaluation. Our method precisely localizes gender representations within middle MLP layers and enables targeted debiasing interventions for both clinical note generation and depression risk prediction; we further demonstrate that although racial representations are distributed across the network, they remain amenable to intervention. These findings establish a novel paradigm for understanding and mitigating structural biases in healthcare AI, offering reproducible, mechanism-grounded technical pathways for equitable model development and deployment.

Technology Category

• Machine Learning: Ethics, Bias, and Fairness
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We know from prior work that LLMs encode social biases, and that this manifests in clinical tasks. In this work we adopt tools from mechanistic interpretability to unveil sociodemographic representations and biases within LLMs in the context of healthcare. Specifically, we ask: Can we identify activations within LLMs that encode sociodemographic information (e.g., gender, race)? We find that gender information is highly localized in middle MLP layers and can be reliably manipulated at inference time via patching. Such interventions can surgically alter generated clinical vignettes for specific conditions, and also influence downstream clinical predictions which correlate with gender, e.g., patient risk of depression. We find that representation of patient race is somewhat more distributed, but can also be intervened upon, to a degree. To our knowledge, this is the first application of mechanistic interpretability methods to LLMs for healthcare.

Problem

• Identify sociodemographic biases in LLMs
• Manipulate gender information in MLP layers
• Intervene on patient race representations

Innovation

• Mechanistic interpretability for LLMs
• Gender localized in MLP layers
• Race representations intervention possible