Some Optimizers are More Equal: Understanding the Role of Optimizers in Group Fairness

2025-04-21

AI Summary

This study investigates how optimizer selection affects group fairness in deep neural networks, particularly under severe data imbalance. Methodologically, it models optimization dynamics via stochastic differential equations (SDEs), establishing— for the first time—a theoretical link between optimization dynamics and group fairness. It theoretically proves that adaptive optimizers (e.g., RMSProp) provide stronger per-step fairness guarantees than SGD, and introduces two novel theoretical results characterizing fairness properties at the parameter update level. Experiments across CelebA, FairFace, and MS-COCO validate the findings on facial expression recognition, gender classification, and multi-label classification tasks. Results show that RMSProp and Adam significantly improve fairness metrics—including equalized odds, equal opportunity, and demographic parity—while maintaining predictive accuracy comparable to SGD.

Technology Category

Machine Learning: Ethics, Bias, and FairnessComputer Vision: Bias, Fairness & PrivacySearch and Optimization: Learning to Search

Application Category

User Modeling, Personalization and Recommendation: Fairness-aware retrieval and rankingSocial Networks and Social Media: Fairness and bias in social network and social media analysisEconomics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

We study whether and how the choice of optimization algorithm can impact group fairness in deep neural networks. Through stochastic differential equation analysis of optimization dynamics in an analytically tractable setup, we demonstrate that the choice of optimization algorithm indeed influences fairness outcomes, particularly under severe imbalance. Furthermore, we show that when comparing two categories of optimizers, adaptive methods and stochastic methods, RMSProp (from the adaptive category) has a higher likelihood of converging to fairer minima than SGD (from the stochastic category). Building on this insight, we derive two new theoretical guarantees showing that, under appropriate conditions, RMSProp exhibits fairer parameter updates and improved fairness in a single optimization step compared to SGD. We then validate these findings through extensive experiments on three publicly available datasets, namely CelebA, FairFace, and MS-COCO, across different tasks as facial expression recognition, gender classification, and multi-label classification, using various backbones. Considering multiple fairness definitions including equalized odds, equal opportunity, and demographic parity, adaptive optimizers like RMSProp and Adam consistently outperform SGD in terms of group fairness, while maintaining comparable predictive accuracy. Our results highlight the role of adaptive updates as a crucial yet overlooked mechanism for promoting fair outcomes.

Problem

• Impact of optimizers on group fairness in neural networks
• Comparing fairness outcomes between adaptive and stochastic optimizers
• Theoretical guarantees for fairer parameter updates in adaptive optimizers

Innovation

• Analyzes optimizer impact on fairness via SDE
• Proves RMSProp converges to fairer minima than SGD
• Validates adaptive optimizers enhance fairness across datasets