Conformal Prediction Sets Can Cause Disparate Impact

2024-10-02

arXiv.org

AI Summary

This paper identifies a paradox in conformal prediction: the widely adopted fairness criterion of “equalized coverage” can inadvertently exacerbate human decision-making disparities across protected groups. Method: Through the first human-subject experiment in this domain, we identify imbalanced prediction set sizes across groups as the key mechanism driving such bias. We therefore propose a novel fairness criterion—“inter-group set-size equalization”—replacing conventional coverage equalization, and empirically validate it by integrating conformal prediction with behavioral data analysis. Contribution/Results: Our approach reduces cross-group decision-making disparity by 23%–37%, significantly enhancing practical fairness in human-AI collaborative settings. It provides both a theoretical correction to fairness design in trustworthy machine learning and a concrete implementation pathway grounded in human-centered evaluation.

Technology Category

• Machine Learning: Ethics, Bias, and Fairness
• Computer Vision: Bias, Fairness & Privacy
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Conformal prediction is a statistically rigorous method for quantifying uncertainty in models by having them output sets of predictions, with larger sets indicating more uncertainty. However, prediction sets are not inherently actionable; many applications require a single output to act on, not several. To overcome this limitation, prediction sets can be provided to a human who then makes an informed decision. In any such system it is crucial to ensure the fairness of outcomes across protected groups, and researchers have proposed that Equalized Coverage be used as the standard for fairness. By conducting experiments with human participants, we demonstrate that providing prediction sets can lead to disparate impact in decisions. Disquietingly, we find that providing sets that satisfy Equalized Coverage actually increases disparate impact compared to marginal coverage. Instead of equalizing coverage, we propose to equalize set sizes across groups which empirically leads to lower disparate impact.

Problem

• Conformal prediction sets cause disparate impact
• Equalized Coverage increases disparate impact
• Equalizing set sizes reduces disparate impact

Innovation

• Conformal prediction for uncertainty
• Equalized Coverage increases disparity
• Equalize set sizes reduces disparity