SHaSaM: Submodular Hard Sample Mining for Fair Facial Attribute Recognition

2026-02-05

AI Summary

This work addresses the issue of prediction unfairness in deep neural networks for facial attribute recognition, which arises from data imbalance and the influence of sensitive attributes such as race, gender, and age. The authors propose SHaSaM, the first method to integrate submodular optimization into fairness-aware learning. SHaSaM employs a two-stage framework: it first selects hard examples via a submodular function to mitigate data imbalance (SHaSaM-MINE), then refines the decision boundary and suppresses sensitivity to protected attributes using a loss function based on submodular conditional mutual information (SHaSaM-LEARN). Experiments on CelebA and UTKFace demonstrate state-of-the-art performance, with up to a 2.7-point improvement in Equalized Odds, a 3.5% gain in accuracy, and faster convergence, effectively balancing fairness and predictive performance.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Machine Learning: Ethics, Bias, and Fairness
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Deep neural networks often inherit social and demographic biases from annotated data during model training, leading to unfair predictions, especially in the presence of sensitive attributes like race, age, gender etc. Existing methods fall prey to the inherent data imbalance between attribute groups and inadvertently emphasize on sensitive attributes, worsening unfairness and performance. To surmount these challenges, we propose SHaSaM (Submodular Hard Sample Mining), a novel combinatorial approach that models fairness-driven representation learning as a submodular hard-sample mining problem. Our two-stage approach comprises of SHaSaM-MINE, which introduces a submodular subset selection strategy to mine hard positives and negatives - effectively mitigating data imbalance, and SHaSaM-LEARN, which introduces a family of combinatorial loss functions based on Submodular Conditional Mutual Information to maximize the decision boundary between target classes while minimizing the influence of sensitive attributes. This unified formulation restricts the model from learning features tied to sensitive attributes, significantly enhancing fairness without sacrificing performance. Experiments on CelebA and UTKFace demonstrate that SHaSaM achieves state-of-the-art results, with up to 2.7 points improvement in model fairness (Equalized Odds) and a 3.5% gain in Accuracy, within fewer epochs as compared to existing methods.

Problem

• fairness
• facial attribute recognition
• data imbalance
• sensitive attributes
• unfair predictions

Innovation

• Submodular Optimization
• Hard Sample Mining
• Fairness-aware Learning
• Conditional Mutual Information
• Demographic Bias Mitigation