seeBias: A Comprehensive Tool for Assessing and Visualizing AI Fairness

2025-04-11

AI Summary

Existing AI fairness tools predominantly assess fairness through classification performance disparities, neglecting critical dimensions such as predictive calibration—limiting their suitability for high-stakes domains like healthcare and criminal justice. To address this gap, we introduce *seeBias*, an open-source R package that unifies classification fairness, predictive calibration, and multidimensional performance evaluation within a single framework. It supports comprehensive diagnostics—including confusion matrix analysis, reliability diagrams, inter-group calibration disparity metrics, and fairness heatmaps—with customizable, publication-ready visualizations. Empirical evaluation on criminal justice and clinical datasets demonstrates that *seeBias* effectively uncovers latent biases: models with comparable accuracy may exhibit severe miscalibration across demographic groups, a deficiency invisible to conventional fairness metrics. The tool is publicly available on GitHub; a Python implementation is under development.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Machine Learning: Ethics, Bias, and Fairness
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Fairness in artificial intelligence (AI) prediction models is increasingly emphasized to support responsible adoption in high-stakes domains such as health care and criminal justice. Guidelines and implementation frameworks highlight the importance of both predictive accuracy and equitable outcomes. However, current fairness toolkits often evaluate classification performance disparities in isolation, with limited attention to other critical aspects such as calibration. To address these gaps, we present seeBias, an R package for comprehensive evaluation of model fairness and predictive performance. seeBias offers an integrated evaluation across classification, calibration, and other performance domains, providing a more complete view of model behavior. It includes customizable visualizations to support transparent reporting and responsible AI implementation. Using public datasets from criminal justice and healthcare, we demonstrate how seeBias supports fairness evaluations, and uncovers disparities that conventional fairness metrics may overlook. The R package is available on GitHub, and a Python version is under development.

Problem

• Assessing AI fairness beyond classification disparities
• Evaluating model fairness and predictive performance comprehensively
• Identifying overlooked disparities in AI fairness metrics

Innovation

• R package for comprehensive AI fairness evaluation
• Integrated assessment across classification and calibration
• Customizable visualizations for transparent reporting