Quantifying Fairness in LLMs Beyond Tokens: A Semantic and Statistical Perspective

2025-06-23

AI Summary

Existing evaluation methods for bias in long-text generation by large language models (LLMs) often overlook output variability and deep semantic biases. Method: We propose FiSCo, the first framework to integrate semantic entailment analysis with statistical hypothesis testing at the claim level, formally defining *group counterfactual fairness* for fine-grained semantic fairness quantification. FiSCo decomposes long responses into comparable claim units via semantic decomposition, entailment judgment, and inter-/intra-group similarity modeling—thereby mitigating interference from output stochasticity. Contribution/Results: Experiments on synthetic and human-annotated datasets demonstrate that FiSCo reliably detects latent biases imperceptible to surface-level lexical analysis, outperforming state-of-the-art fairness metrics in both robustness and semantic sensitivity.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Large Language Models (LLMs) often generate responses with inherent biases, undermining their reliability in real-world applications. Existing evaluation methods often overlook biases in long-form responses and the intrinsic variability of LLM outputs. To address these challenges, we propose FiSCo(Fine-grained Semantic Computation), a novel statistical framework to evaluate group-level fairness in LLMs by detecting subtle semantic differences in long-form responses across demographic groups. Unlike prior work focusing on sentiment or token-level comparisons, FiSCo goes beyond surface-level analysis by operating at the claim level, leveraging entailment checks to assess the consistency of meaning across responses. We decompose model outputs into semantically distinct claims and apply statistical hypothesis testing to compare inter- and intra-group similarities, enabling robust detection of subtle biases. We formalize a new group counterfactual fairness definition and validate FiSCo on both synthetic and human-annotated datasets spanning gender, race, and age. Experiments show that FiSco more reliably identifies nuanced biases while reducing the impact of stochastic LLM variability, outperforming various evaluation metrics.

Problem

• Detect subtle semantic biases in LLM long-form responses
• Evaluate group-level fairness beyond token-level analysis
• Address stochastic variability in LLM outputs for bias detection

Innovation

• Fine-grained semantic computation for bias detection
• Claim-level entailment checks for meaning consistency
• Statistical hypothesis testing for group fairness