Medical Imaging AI Competitions Lack Fairness

2025-12-19

AI Summary

Medical imaging AI challenges suffer from two systemic fairness deficiencies: insufficient clinical representativeness and poor FAIR (Findable, Accessible, Interoperable, Reusable) compliance. To address this, we conducted the first large-scale systematic assessment of 241 international challenges, introducing a novel dual-dimensional fairness evaluation framework: (1) quantitative assessment of data representativeness—measuring geographic, modality, and disease diversity; and (2) audit of FAIR adherence—evaluating access policies, license legality, and documentation completeness. Leveraging systematic review, meta-analysis, and cross-modal/geographic statistical modeling, we uncovered pronounced geographic and modality biases; over 60% of datasets impose restrictive or ambiguous access terms; 48% lack compliant licenses, and 49% exhibit incomplete documentation. This study provides the first empirical evidence of widespread fairness deficits in medical imaging challenges, delivering an actionable assessment toolkit and concrete improvement pathways for challenge design, data governance, and clinical translation.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Philosophy and Ethics of AI: Bias, Fairness & Equity
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Benchmarking competitions are central to the development of artificial intelligence (AI) in medical imaging, defining performance standards and shaping methodological progress. However, it remains unclear whether these benchmarks provide data that are sufficiently representative, accessible, and reusable to support clinically meaningful AI. In this work, we assess fairness along two complementary dimensions: (1) whether challenge datasets are representative of real-world clinical diversity, and (2) whether they are accessible and legally reusable in line with the FAIR principles. To address this question, we conducted a large-scale systematic study of 241 biomedical image analysis challenges comprising 458 tasks across 19 imaging modalities. Our findings show substantial biases in dataset composition, including geographic location, modality-, and problem type-related biases, indicating that current benchmarks do not adequately reflect real-world clinical diversity. Despite their widespread influence, challenge datasets were frequently constrained by restrictive or ambiguous access conditions, inconsistent or non-compliant licensing practices, and incomplete documentation, limiting reproducibility and long-term reuse. Together, these shortcomings expose foundational fairness limitations in our benchmarking ecosystem and highlight a disconnect between leaderboard success and clinical relevance.

Problem

• Medical imaging AI competitions lack fairness in dataset representativeness.
• They face accessibility and legal reusability issues under FAIR principles.
• Biases and constraints limit clinical relevance and reproducibility of benchmarks.

Innovation

• Systematically assessing dataset representativeness and FAIR compliance
• Identifying geographic, modality, and problem type biases in benchmarks
• Highlighting restrictive access and licensing as reproducibility barriers