Beyond Third-Person Audits: Situated Interaction Auditing for User-Centered LLM Bias Research

2026-06-10

AI Summary

This study addresses a critical gap in current research on bias in large language models (LLMs), which predominantly relies on third-person auditing and overlooks how users’ identity cues during interaction shape model responses. The authors propose a Situated Interactive Auditing (SIA) framework that places the user at the center of bias evaluation by systematically simulating implicit sociodemographic attributes, writing styles, and explicit identity statements across diverse tasks. Through experiments and case analyses, they demonstrate that user identity significantly influences the quality, content, and tone of LLM outputs. Their findings reveal that bias manifests not only in representations of “others” but also in differential treatment of users based on their perceived identities, thereby establishing a user-centered paradigm for LLM bias research.

Abstract

Research on bias in large language models (LLMs) has predominantly focused on third-person audits, which study how models represent or evaluate demographic groups as external subjects. However, this paradigm overlooks a structural blind spot because the user is absent from the audit. In practice, LLMs are used in open-ended, personal interactions, during which the model implicitly represents the user and adjusts its responses accordingly. When identical requests yield different responses depending on who is asking, bias manifests not in how the model describes others but in how it treats its interlocutor. We propose Situated Interaction Auditing (SIA), a user-centered framework for studying how user profile signals -- implicit sociodemographic markers, writing style, and stated identity -- systematically shape LLM response quality, content, and tone. We demonstrate the framework through a case study that intersects gender and socioeconomic status signals across multiple task domains and outline a research agenda for SIA as a new mission for natural language processing.

Problem

• LLM bias
• user-centered auditing
• situated interaction
• sociodemographic signals
• response variation

Innovation

• Situated Interaction Auditing
• user-centered bias
• large language models
• sociodemographic signals
• interactive fairness