Contrastive Analysis of Linguistic Representations in Large Language Model Outputs through Structured Synthetic Data Generation and Abstracted N-gram Associations

2026-04-19

AI Summary

This work proposes a context-aware bias detection framework that identifies subtle linguistic biases in large language model outputs toward diverse social groups without relying on predefined lists of sensitive terms. The approach generates structured synthetic minimal-pair texts—narratively consistent except for the substitution of target group markers—and employs linguistic form abstraction combined with an enhanced variant of pointwise mutual information (PMI) for comparative analysis. Integrating quantitative statistics with qualitative evaluation, the framework is adaptable across multiple text genres and effectively quantifies asymmetric associations between social groups and levels of linguistic abstraction. It precisely localizes textual segments with high concentrations of bias signals, enabling domain experts to identify potentially harmful expressions within their contextual settings.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness
• Computer Vision: Bias, Fairness & Privacy

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Web Mining and Content Analysis: Sentiment analysis and opinion mining
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

We present a methodological framework to discover linguistic and discursive patterns associated to different social groups through contrastive synthetic text generation and statistical analysis. In contrast with previous approaches, we aim to characterize subtle expressions of bias, instead of diagnosing bias through a pre-determined list of words or expressions. We are also working with contextualized data instead of isolated words or sentences. Our methodology applies to textual productions in any genre, encompassing narrative, task-oriented or dialogic. Contextualized data are generated using controlled combinations of situational scenarios and group markers, creating minimal pairs of texts that differ only in the referenced group while maintaining comparable narrative conditions. To facilitate robust analysis, linguistic forms are generalized and associations between linguistic abstractions and groups are quantified using a variant of pointwise mutual information to detect expressions that appear disproportionately across groups. A fragment-ranking strategy then prioritizes text segments with a high concentration of biased linguistic signals, which allows for experts to assess the harmful potential of linguistic expressions in context, bridging quantitative analysis and qualitative interpretation.

Problem

• linguistic bias
• large language models
• contrastive analysis
• synthetic data
• contextualized representations

Innovation

• contrastive synthetic data
• contextualized bias detection
• linguistic abstraction
• pointwise mutual information variant
• minimal pair generation