AI-Face: A Million-Scale Demographically Annotated AI-Generated Face Dataset and Fairness Benchmark

2024-06-02

arXiv.org

AI Summary

Current AI-generated face detectors exhibit significant demographic bias, primarily due to the absence of large-scale benchmark datasets that jointly provide fine-grained demographic annotations (e.g., gender, age, race) and diverse generative methods (e.g., Deepfakes, GANs, diffusion models). To address this, we introduce FairFaceGen—the first million-scale, dual-dimensionally unified AI face dataset—systematically aligning generative diversity with high-fidelity demographic labeling. We further propose a bias quantification evaluation framework and establish the first open-source benchmark explicitly designed for fairness-aware detection. Empirical analysis reveals performance disparities of 15–30% across demographic groups in state-of-the-art detectors. All components—including the dataset, source code, and evaluation protocols—are publicly released, enabling reproducible assessment and fostering principled advances in fair AI-based face detection.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Philosophy and Ethics of AI: Bias, Fairness & Equity
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems

Abstract

AI-generated faces have enriched human life, such as entertainment, education, and art. However, they also pose misuse risks. Therefore, detecting AI-generated faces becomes crucial, yet current detectors show biased performance across different demographic groups. Mitigating biases can be done by designing algorithmic fairness methods, which usually require demographically annotated face datasets for model training. However, no existing dataset encompasses both demographic attributes and diverse generative methods simultaneously, which hinders the development of fair detectors for AI-generated faces. In this work, we introduce the AI-Face dataset, the first million-scale demographically annotated AI-generated face image dataset, including real faces, faces from deepfake videos, and faces generated by Generative Adversarial Networks and Diffusion Models. Based on this dataset, we conduct the first comprehensive fairness benchmark to assess various AI face detectors and provide valuable insights and findings to promote the future fair design of AI face detectors. Our AI-Face dataset and benchmark code are publicly available at https://github.com/Purdue-M2/AI-Face-FairnessBench

Problem

• Detecting AI-generated faces with demographic bias
• Lack of datasets with demographic and generative diversity
• Developing fair AI face detectors using annotated datasets

Innovation

• Million-scale demographically annotated AI-generated dataset
• Includes diverse generative methods and real faces
• First comprehensive fairness benchmark for AI face detectors