Detecting Linguistic Bias in Government Documents Using Large language Models

2025-02-19

AI Summary

This study addresses the long-overlooked yet critical problem of detecting implicit linguistic bias in government documents—a key concern for equitable governance. Existing methods suffer from poor domain adaptability and fail to uncover the mechanisms through which bias operates within policy contexts. To bridge this gap, we introduce DGDB, the first expert-annotated bias dataset specifically designed for Dutch parliamentary documents, covering fine-grained bias types across public policy domains. Methodologically, we integrate domain-adapted BERT-based fine-tuning, zero- and few-shot inference with generative large language models, and expert-driven annotation coupled with interpretable error analysis. Experimental results demonstrate that domain-finetuned BERT models substantially outperform general-purpose LLMs on bias detection. DGDB serves as the first benchmark resource enabling reproducible, multilingual research on governance fairness, providing both foundational data and an extensible methodological framework for fairness assessment in governmental texts.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

This paper addresses the critical need for detecting bias in government documents, an underexplored area with significant implications for governance. Existing methodologies often overlook the unique context and far-reaching impacts of governmental documents, potentially obscuring embedded biases that shape public policy and citizen-government interactions. To bridge this gap, we introduce the Dutch Government Data for Bias Detection (DGDB), a dataset sourced from the Dutch House of Representatives and annotated for bias by experts. We fine-tune several BERT-based models on this dataset and compare their performance with that of generative language models. Additionally, we conduct a comprehensive error analysis that includes explanations of the models' predictions. Our findings demonstrate that fine-tuned models achieve strong performance and significantly outperform generative language models, indicating the effectiveness of DGDB for bias detection. This work underscores the importance of labeled datasets for bias detection in various languages and contributes to more equitable governance practices.

Problem

• Detecting bias in government documents
• Using large language models for bias detection
• Improving governance through equitable practices

Innovation

• Fine-tuned BERT-based models
• Dutch Government Data for Bias Detection
• Outperform generative language models