Counterfactual Fairness with Graph Uncertainty

2026-01-06

arXiv.org

AI Summary

This work addresses the limitation of existing counterfactual fairness assessments, which typically assume a single fixed causal graph and thereby overlook the inherent uncertainty in causal structures within real-world scenarios, leading to potentially unreliable audit conclusions. To remedy this, the authors propose CF-GU, a novel framework that systematically incorporates causal graph uncertainty into counterfactual fairness analysis. Under domain knowledge constraints, CF-GU employs causal discovery algorithms combined with bootstrap resampling to generate an ensemble of plausible directed acyclic graphs (DAGs) and computes confidence intervals for fairness metrics across this DAG ensemble. The method further introduces normalized Shannon entropy to quantify graph uncertainty, enabling confidence-aware fairness auditing. Experiments demonstrate that CF-GU reveals the influence of domain knowledge on fairness judgments in synthetic data and reliably identifies known biases in the COMPAS and Adult datasets with high confidence, even under minimal prior constraints.

Technology Category

• Reasoning under Uncertainty: Causality
• Machine Learning: Causal Learning
• Computer Vision: Bias, Fairness & Privacy

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Evaluating machine learning (ML) model bias is key to building trustworthy and robust ML systems. Counterfactual Fairness (CF) audits allow the measurement of bias of ML models with a causal framework, yet their conclusions rely on a single causal graph that is rarely known with certainty in real-world scenarios. We propose CF with Graph Uncertainty (CF-GU), a bias evaluation procedure that incorporates the uncertainty of specifying a causal graph into CF. CF-GU (i) bootstraps a Causal Discovery algorithm under domain knowledge constraints to produce a bag of plausible Directed Acyclic Graphs (DAGs), (ii) quantifies graph uncertainty with the normalized Shannon entropy, and (iii) provides confidence bounds on CF metrics. Experiments on synthetic data show how contrasting domain knowledge assumptions support or refute audits of CF, while experiments on real-world data (COMPAS and Adult datasets) pinpoint well-known biases with high confidence, even when supplied with minimal domain knowledge constraints.

Problem

• Counterfactual Fairness
• Graph Uncertainty
• Causal Graph
• Bias Evaluation
• Machine Learning Fairness

Innovation

• Counterfactual Fairness
• Graph Uncertainty
• Causal Discovery
• Directed Acyclic Graphs
• Bias Evaluation