Is invariance all you need for algorithmic fairness? Removing demographic information can create new bias

2026-09-25

AI Summary

This study addresses the prevalent assumption in existing research that demographic invariance in representations serves as a valid criterion for mitigating algorithmic bias, an assertion that has historically lacked rigorous theoretical grounding. By formally distinguishing between marginal and class-conditional invariance, this work demonstrates through mathematical derivation that neither constitutes a sufficient nor necessary condition for algorithmic fairness. These theoretical findings are further corroborated by empirical validation across multimodal datasets, including tabular data and medical imaging. Ultimately, this research establishes from both theoretical and experimental perspectives that enforcing representational invariance not only fails to guarantee fairness but may also impede debiasing efforts and introduce novel biases. The core contribution lies in demonstrating that demographic representational invariance is neither sufficient nor desirable for achieving algorithmic fairness.

Abstract

Encoded demographic information in internal model representations is a commonly assumed risk factor for algorithmic bias, with demographic representation invariance often being touted as the ideal state. However, while demographic shortcut learning is a genuine threat, some degree of encoding is necessary when demographics correlate with target labels. Here, we show, mathematically and empirically, that enforcing demographic invariance can actually hamper bias mitigation and even create new biases. We distinguish marginal from class-conditional representation invariance, and show that they imply the standard group fairness notions of demographic parity and equalized odds, respectively. We evaluate the effects on predictive performance and fairness of enforcing both invariance types, both theoretically and empirically across five tabular and two chest X-ray imaging datasets. Our findings support our mathematical argument that demographic representation invariance is neither desirable nor sufficient for fairness.

Problem

• algorithmic fairness
• demographic invariance
• representation bias
• shortcut learning
• group fairness

Innovation

• Algorithmic Fairness
• Demographic Invariance
• Representation Learning
• Bias Mitigation
• Equalized Odds