subgroup analysis

This paper identifies a critical issue: when training and test data are drawn from the same source via random sampling, inherent data biases systematically distort fairness evaluation, leading to erroneous assessments for protected subgroups (e.g., gender, race). To address this, we propose FairMatch—the first fairness diagnostic method integrating Propensity Score Matching (PSM). FairMatch constructs comparable inter-subgroup sample pairs on the test set, dynamically optimizes subgroup-specific decision thresholds, and applies fairness-aware probability calibration to unmatched samples. By enabling precise bias localization and hierarchical mitigation, FairMatch preserves model predictive performance while significantly enhancing the reliability of fairness assessment and the effectiveness of bias mitigation.

In large-scale online platforms with hundreds of millions of users, conventional A/B testing is infeasible for post-hoc policy evaluation under sparse interventions, leading to severe estimation bias. To address this, we propose a two-stage debiased counterfactual estimation framework: (1) a covariate-based nearest-neighbor matching stage to construct high-fidelity control units and mitigate interpolation bias; and (2) a high-dimensional supervised learning stage—using XGBoost or neural networks—to model treatment effects while systematically diagnosing and correcting machine-learning-induced estimation bias. This is the first scalable solution enabling synthetic control methods to operate effectively under ultra-large-scale, sparse-intervention settings. Evaluated across six real-world online experiments, our method significantly improves causal effect estimation accuracy and reduces policy decision error rates by 42%. It has been deployed operationally to support closed-loop decision-making across multiple business units.

This paper addresses the misalignment between algorithmic bias assessment and legal standards by proposing a quantification framework rigorously grounded in U.S. anti-discrimination law. Methodologically, it distinguishes legally salient discriminatory testing from systemic disparity through legal contextualization, and introduces the Objective Fairness Index (OFI)—a metric integrating objective test theory and measurement stability, using marginal benefit as a proxy to quantify legal compliance of algorithmic decisions. Its key contribution lies in being the first fairness metric to embed legal admissibility directly into its design, enabling a paradigm shift in algorithmic auditing from statistical fairness to legally grounded fairness. Empirical evaluation on real-world judicial prediction systems—including COMPAS—demonstrates that OFI reliably detects unlawful discrimination, offering regulators and auditors the first quantitative tool with both legal interpretability and operational utility.

Machine learning models exhibit high sensitivity to minor perturbations in training data, leading to unstable predictions; yet conventional fairness metrics (e.g., bias-based indicators) ignore this prediction uncertainty. Method: We propose a variance-oriented paradigm for group fairness—introducing the first systematic framework that treats inter-group predictive variance equality as a core fairness criterion, grounded in statistical error decomposition and theoretical analysis of variance’s independent impact on fairness assessment. Contribution/Results: We release VarFair, the first open-source library integrating uncertainty quantification with fairness evaluation. Extensive experiments on Adult, COMPAS, and other benchmarks demonstrate that groups with high predictive variance are frequently misclassified as “fair” by standard methods, whereas our variance-aware metric significantly improves identification of disadvantaged groups and enhances assessment robustness under data perturbations.

This study addresses two fundamental challenges in evaluating individualized treatment benefit predictors (TBPs) from observational data: nonidentifiability and confounding bias. Methodologically, we first establish that confounding bias propagates nonlinearly and unpredictably in TBP evaluation; we then develop a novel identifiability framework grounded solely in observable data, leveraging latent-variable reconstruction—including the benefit concentration index and moderate calibration curve—to derive causal identifiability expressions for discrimination and calibration metrics. We theoretically prove identifiability under partial confounder control and quantify the systematic failure of conventional causal intuition in this setting. Our contributions provide a new paradigm and practical toolkit for robust TBP evaluation in clinical decision support.

This study addresses the inconsistency among fairness metrics in face recognition, where different measures often yield contradictory conclusions about model bias, thereby exposing the limitations of single-metric evaluation. To tackle this issue, the authors propose the Fairness Disagreement Index (FDI) to quantify the degree of disagreement across multiple fairness criteria and introduce a multidimensional evaluation framework that integrates both error rate disparities and performance-oriented fairness metrics. Through systematic experiments under controlled conditions, they demonstrate that such metric disagreement is pervasive across varying decision thresholds and model configurations, revealing a critical flaw in current fairness assessment practices. The work provides both a novel analytical tool and empirical evidence to support more comprehensive and reliable fairness evaluations in face recognition systems.

This work addresses the susceptibility of existing calibration evaluations for large language models to accuracy disparities, which distorts cross-model comparisons. To enable fair calibration assessment while controlling for accuracy, the authors propose the ACE framework, incorporating three alignment mechanisms: instance alignment, distribution alignment, and candidate alignment. The study systematically reveals, for the first time, the bias inherent in conventional global calibration metrics—such as Expected Calibration Error and Brier Score—when used for comparing models with differing accuracies, and introduces an accuracy-controlled correction strategy. Experimental results demonstrate that the apparent calibration advantages of most models substantially diminish—and their rankings frequently reverse—once calibration metrics are adjusted for accuracy, thereby demonstrating that unadjusted metrics are unsuitable for cross-model calibration evaluation.

This study addresses the disconnect between academic research and industrial practice in treatment effect estimation, where prevailing evaluation paradigms hinder real-world applicability. Through a large-scale empirical analysis, we systematically compare diverse meta-learners, base learners, and specialized causal models across semi-synthetic benchmarks and real-world datasets. Our findings reveal a pronounced inconsistency between counterfactual and observable performance metrics, and demonstrate that model rankings derived from semi-synthetic data fail to generalize to real settings. Notably, simple meta-learners paired with strong base models consistently outperform purpose-built causal models on real data, underscoring the critical importance of validation on real-world outcomes and observable metrics. These results challenge the dominant reliance on semi-synthetic evaluations and call for a paradigm shift toward more empirically grounded assessment protocols.

This study addresses how infra-marginality—differences in data distributions across groups—complicates judgments of AI fairness, as conventional statistical parity metrics often fail to align with human perceptions of fairness. Through a controlled user study involving 85 participants in a hypothetical medical decision-making scenario, the authors systematically investigate how group-specific model performance and training data availability shape fairness judgments. They find that when group-wise performance is equal or unknown, participants favor outcome equality; however, when performance disparities are attributable to data imbalance, models preserving these differences are perceived as more fair. These results demonstrate that human fairness judgments are not solely based on outcome equality but are significantly influenced by beliefs about the underlying causes of disparities, thereby challenging the prevailing assumption that statistical parity should serve as the default standard for algorithmic fairness.

This study addresses fairness in toxicity detection through a multidimensional lens encompassing ranking, calibration, and rejection. It systematically evaluates the interplay between training interventions—empirical risk minimization (ERM), sample reweighting, and Group Distributionally Robust Optimization (Group DRO)—and post-processing mechanisms, including temperature scaling, confidence-based rejection, and identity-specific thresholds. The findings reveal that miscalibration constitutes a subtle yet critical form of fairness violation; current training methods reshape rather than eliminate disparities; and rejection strategies can inadvertently exacerbate inter-group inequities. Specifically, ERM achieves good global calibration but exhibits severe subgroup miscalibration, reweighting improves ranking at the cost of calibration fairness, and Group DRO eliminates calibration gaps while inducing overall miscalibration. Confidence-based rejection proves effective under ERM but fails under DRO, disproportionately disadvantaging content mentioning specific identities. The work advocates for a joint multi-axis evaluation framework to holistically assess fairness.