Gender-Neutral Large Language Models for Medical Applications: Reducing Bias in PubMed Abstracts

2025-01-10

AI Summary

This study addresses gender bias in medical large language models by proposing MOBERT, the first de-gendered pretraining framework tailored for PubMed literature. Methodologically, it leverages 379,000 historical medical abstracts (1965–1980) to construct a profession-agnostic pronoun neutralization pipeline: custom pronoun identification and substitution, historical text cleaning, neutral annotation, and supervised fine-tuning—integrated directly into the BERT pretraining stage via neutralized pronoun embeddings. Experiments show MOBERT achieves 70% accuracy on pronoun neutralization, substantially outperforming the baseline 1965Bert (4%); moreover, term frequency exhibits a strong positive correlation with correction accuracy. This work constitutes the first systematic effort toward pronoun debiasing in medical pretraining, establishing a reproducible methodological paradigm for fair and inclusive medical language modeling.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Ethics, Bias, and Fairness
• Computer Vision: Bias, Fairness & Privacy

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This paper presents a pipeline for mitigating gender bias in large language models (LLMs) used in medical literature by neutralizing gendered occupational pronouns. A dataset of 379,000 PubMed abstracts from 1965-1980 was processed to identify and modify pronouns tied to professions. We developed a BERT-based model, ``Modern Occupational Bias Elimination with Refined Training,'' or ``MOBERT,'' trained on these neutralized abstracts, and compared its performance with ``1965Bert,'' trained on the original dataset. MOBERT achieved a 70% inclusive replacement rate, while 1965Bert reached only 4%. A further analysis of MOBERT revealed that pronoun replacement accuracy correlated with the frequency of occupational terms in the training data. We propose expanding the dataset and refining the pipeline to improve performance and ensure more equitable language modeling in medical applications.

Problem

• Gender Bias
• Medical Literature
• Large Language Models

Innovation

• MOBERT
• Gender-neutral
• Medical Language Modeling