Debias Anything: Fairness with Diversity without Supervision in Diffusion Models

2026-10-01

AI Summary

This study addresses the tendency of diffusion models to amplify demographic biases present in training data, noting that existing debiasing methods often compromise diversity or rely on sensitive attribute annotations. To overcome these limitations, this work proposes an unsupervised joint optimization framework that maps a frozen diffusion model into a pretrained vision-language embedding space via adapters. It leverages text pairs to define attribute directions for guiding batch composition and introduces a semantic divergence-based diversity scoring mechanism. This approach is the first to simultaneously achieve fairness and diversity without requiring sensitive attribute labels. Experimental results demonstrate that the proposed method significantly enhances both the quality and diversity of generated images under equivalent fairness constraints, while generalizing effectively across arbitrary diffusion models and sensitive attributes.

Abstract

Although diffusion models produce high-quality images, they also reproduce and amplify demographic imbalances in their training data. Debiasing their generation process post-training w.r.t. some sensitive attribute usually relies on classifier guidance or explicit text extra-conditioning, but this reduces methods' applicability and output diversity. Conversely, methods promoting diversity alone do not ensure fair attribute representation. In this paper, we propose a method tackling fairness and diversity jointly that is generally applicable to any diffusion model and any sensitive attribute. To this end, an adapter connects the frozen diffusion model to a pretrained vision-language embedding space, enabling fairness and diversity guidance without sensitive-attribute annotations. For fairness, pairs of text prompts define attribute directions which guide batch composition towards specific proportions. For diversity, we introduce a score measuring disagreement between the semantic estimates derived from this representation. The formulation supports unconditional and text-conditional diffusion models, while requiring no prior knowledge or data of sensitive attribute. Experiments confirm that our method improves quality and diversity scores at comparable fairness levels.

Problem

• Diffusion Models
• Fairness
• Diversity
• Debiasing
• Unsupervised

Innovation

• Diffusion Models
• Debiasing
• Fairness
• Diversity
• Vision-Language Embedding