Biased Heritage: How Datasets Shape Models in Facial Expression Recognition

2025-03-05

AI Summary

This paper investigates how dataset bias propagates through intersecting demographic dimensions—such as gender, race, and age—to model predictions in facial expression recognition (FER), causing multi-class fairness disparities. Existing fairness studies predominantly focus on binary classification and overlook the intrinsic multi-class and multi-group interaction characteristics of FER. Method: We propose the first fine-grained, multi-group bias measurement framework for FER, explicitly distinguishing representational bias from emotion-specific stereotypical bias. Our methodology integrates controlled bias injection, multi-group fairness evaluation, bias–performance modeling, and attribution analysis. Contribution/Results: We find that emotion–demographic interaction bias exhibits higher “contagiousness” across groups; mitigating it simultaneously improves both fairness and accuracy—challenging the conventional fairness–accuracy trade-off assumption. Moreover, intervening in bias propagation pathways proves more effective than merely balancing demographic distributions in training data.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Machine Learning: Ethics, Bias, and Fairness
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

In recent years, the rapid development of artificial intelligence (AI) systems has raised concerns about our ability to ensure their fairness, that is, how to avoid discrimination based on protected characteristics such as gender, race, or age. While algorithmic fairness is well-studied in simple binary classification tasks on tabular data, its application to complex, real-world scenarios-such as Facial Expression Recognition (FER)-remains underexplored. FER presents unique challenges: it is inherently multiclass, and biases emerge across intersecting demographic variables, each potentially comprising multiple protected groups. We present a comprehensive framework to analyze bias propagation from datasets to trained models in image-based FER systems, while introducing new bias metrics specifically designed for multiclass problems with multiple demographic groups. Our methodology studies bias propagation by (1) inducing controlled biases in FER datasets, (2) training models on these biased datasets, and (3) analyzing the correlation between dataset bias metrics and model fairness notions. Our findings reveal that stereotypical biases propagate more strongly to model predictions than representational biases, suggesting that preventing emotion-specific demographic patterns should be prioritized over general demographic balance in FER datasets. Additionally, we observe that biased datasets lead to reduced model accuracy, challenging the assumed fairness-accuracy trade-off.

Problem

• Analyzing bias propagation in Facial Expression Recognition systems.
• Developing bias metrics for multiclass problems with multiple demographics.
• Challenging the fairness-accuracy trade-off in biased datasets.

Innovation

• Introduces new bias metrics for multiclass FER
• Analyzes bias propagation from datasets to models
• Challenges fairness-accuracy trade-off in biased datasets