Beyond RMSE and MAE: Introducing EAUC to unmask hidden bias and unfairness in dyadic regression models

2024-01-19

arXiv.org

AI Summary

In binary regression models, uneven entity observation frequencies induce “eccentricity bias”—a latent fairness issue where predictions systematically regress toward historical means, underperforming even random guessing on tail entities, and remaining undetected by conventional metrics (e.g., RMSE, MAE). To address this, we propose EAUC (Eccentricity-AUC), the first dedicated metric for quantifying such bias. Our approach integrates theoretical modeling of eccentricity bias, empirical sensitivity analysis, lightweight post-training calibration, and a fairness-aware modeling guidance framework—establishing a novel bias-aware evaluation paradigm. Validated across diverse domains including recommender systems and personalized pharmacology, EAUC effectively exposes hidden unfairness in state-of-the-art models. It enables more robust and equitable modeling practices by surfacing previously obscured performance disparities across entity frequency strata.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Philosophy and Ethics of AI: Bias, Fairness & Equity
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Social Networks and Social Media: Fairness and bias in social network and social media analysis

Abstract

Dyadic regression models, which output real-valued predictions for pairs of entities, are fundamental in many domains (e.g. obtaining user-product ratings in Recommender Systems) and promising and under exploration in others (e.g. tuning patient-drug dosages in personalized pharmacology). In this work, we prove that non-uniform observed value distributions of individual entities lead to severe biases in state-of-the-art models, skewing predictions towards the average of observed past values for the entity and providing worse-than-random predictive power in eccentric yet crucial cases; we name this phenomenon eccentricity bias. We show that global error metrics like Root Mean Squared Error (RMSE) are insufficient to capture this bias, and we introduce Eccentricity-Area Under the Curve (EAUC) as a novel complementary metric that can quantify it in all studied domains and models. We prove the intuitive interpretation of EAUC by experimenting with naive post-training bias corrections, and theorize other options to use EAUC to guide the construction of fair models. This work contributes a bias-aware evaluation of dyadic regression to prevent unfairness in critical real-world applications of such systems.

Problem

• Dyadic Prediction Models
• Eccentric Bias
• Evaluation Metrics

Innovation

• EAUC
• Bias Detection
• Model Fairness