Scaling Vision-Language Models Is Not Enough to Mitigate Bias

2026-07-30

AI Summary

This study addresses the limited robustness of large vision-language models against spurious correlations—such as single- and multi-attribute biases—and the unclear relationship between model scale and debiasing efficacy. Through a systematic empirical analysis of 194 publicly available models, the authors evaluate how model size, training data, and architectural design influence bias sensitivity on ImageNet, CelebA, and UrbanCars benchmarks. They find that increasing model scale has nearly no effect on mitigating complex multi-attribute biases (ρ = 0.05), whereas high-quality, large-scale training data consistently improves worst-group accuracy by up to 25%. The impact of architectural choices, however, is highly dependent on the specific bias type and its spatial distribution.

Abstract

Vision-Language Models (VLMs) such as CLIP are now foundational to multimodal systems, yet their robustness to spurious correlations remains poorly understood at scale. We present the first large-scale empirical study of 194 publicly available VLMs, including 16 model families, covering a wide range of model sizes, 24 training datasets, and three evaluation benchmarks, namely ImageNet (overall performance), CelebA (typical single-attribute bias), and UrbanCars (complex multi-attribute biases). Across these settings, the Spearman correlation between model scale and performance weakens as evaluation shifts from ImageNet ($ρ{=}0.68$) to single-attribute ($ρ{=}0.48$) and further to multi-attribute ($ρ{=}0.05$) bias benchmarks. In contrast, properties of the training data (size and quality) show more consistent relationships with worst-group accuracy across both bias benchmarks. Notably, curated datasets yield improvements of up to 25% over uncurated alternatives at a comparable scale. Finally, the effect of architectural choices (e.g., patch size, image resolution) is highly context-dependent, varying with the nature of the benchmark, including the type of bias and its spatial distribution within images.

Problem

• Vision-Language Models
• Bias Mitigation
• Spurious Correlations
• Model Scaling
• Robustness

Innovation

• vision-language models
• bias mitigation
• training data quality
• model scaling
• worst-group accuracy