Decomposing Global AUC into Cluster-Level Contributions for Localized Model Diagnostics

2025-08-10

AI Summary

Global AUC fails to reveal localized model performance deficiencies within subpopulations. To address this, we propose the first decomposable cluster-level AUC evaluation framework, which rigorously decomposes global AUC into two orthogonal components: intra-cluster ranking ability and inter-cluster discriminative ability. Methodologically, leveraging the geometric properties of ROC curves and cluster structure, we derive an additive, unbiased AUC decomposition formula and provide theoretical guarantees for its interpretability and statistical consistency. Unlike conventional metrics such as Brier score or log loss, our framework is the first to enable AUC decomposition at the cluster granularity, facilitating fine-grained diagnostic analysis and high-risk subgroup identification. Empirical evaluations on credit approval and fraud detection tasks demonstrate significant improvements in model validation accuracy and risk management efficacy.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Search and Optimization: Distributed Search

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

The Area Under the ROC Curve (AUC) is a widely used performance metric for binary classifiers. However, as a global ranking statistic, the AUC aggregates model behavior over the entire dataset, masking localized weaknesses in specific subpopulations. In high-stakes applications such as credit approval and fraud detection, these weaknesses can lead to financial risk or operational failures. In this paper, we introduce a formal decomposition of global AUC into intra- and inter-cluster components. This allows practitioners to evaluate classifier performance within and across clusters of data, enabling granular diagnostics and subgroup analysis. We also compare the AUC with additive performance metrics such as the Brier score and log loss, which support decomposability and direct attribution. Our framework enhances model development and validation practice by providing additional insights to detect model weakness for model risk management.

Problem

• Decompose global AUC for localized model diagnostics
• Evaluate classifier performance within and across clusters
• Compare AUC with decomposable metrics like Brier score

Innovation

• Decomposes global AUC into cluster-level components
• Enables granular diagnostics via intra-inter cluster analysis
• Compares AUC with decomposable additive performance metrics