Fairness Evaluation of Large Language Models in Academic Library Reference Services

2025-07-05

AI Summary

This study addresses the lack of systematic evaluation of fairness in large language models’ (LLMs) responses within academic library virtual reference services—specifically across gender, race/ethnicity, and institutional affiliation (e.g., student, librarian, faculty). Six state-of-the-art LLMs were prompted under controlled conditions to generate responses to identical reference queries; linguistic analyses quantified disparities in formality, politeness, and domain-specific terminology across user identity dimensions. Results reveal no statistically significant racial/ethnic bias; only one model exhibited mild stereotyping toward female users. Overall, responses adhered to professional reference service norms, demonstrating strong contextual appropriateness and equitable treatment across identities. This work constitutes the first multidimensional fairness assessment of LLMs in library contexts, bridging a critical gap in the literature and empirically validating their potential to support inclusive, identity-aware information services.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Social Networks and Social Media: Fairness and bias in social network and social media analysis

Abstract

As libraries explore large language models (LLMs) for use in virtual reference services, a key question arises: Can LLMs serve all users equitably, regardless of demographics or social status? While they offer great potential for scalable support, LLMs may also reproduce societal biases embedded in their training data, risking the integrity of libraries' commitment to equitable service. To address this concern, we evaluate whether LLMs differentiate responses across user identities by prompting six state-of-the-art LLMs to assist patrons differing in sex, race/ethnicity, and institutional role. We found no evidence of differentiation by race or ethnicity, and only minor evidence of stereotypical bias against women in one model. LLMs demonstrated nuanced accommodation of institutional roles through the use of linguistic choices related to formality, politeness, and domain-specific vocabularies, reflecting professional norms rather than discriminatory treatment. These findings suggest that current LLMs show a promising degree of readiness to support equitable and contextually appropriate communication in academic library reference services.

Problem

• Evaluate fairness of LLMs in library reference services
• Assess bias in LLM responses across demographics
• Determine if LLMs accommodate institutional roles equitably

Innovation

• Evaluating LLM fairness across diverse user identities
• Testing six state-of-the-art models for bias
• Analyzing linguistic choices for institutional role accommodation