roc analysis

Global AUC fails to reveal localized model performance deficiencies within subpopulations. To address this, we propose the first decomposable cluster-level AUC evaluation framework, which rigorously decomposes global AUC into two orthogonal components: intra-cluster ranking ability and inter-cluster discriminative ability. Methodologically, leveraging the geometric properties of ROC curves and cluster structure, we derive an additive, unbiased AUC decomposition formula and provide theoretical guarantees for its interpretability and statistical consistency. Unlike conventional metrics such as Brier score or log loss, our framework is the first to enable AUC decomposition at the cluster granularity, facilitating fine-grained diagnostic analysis and high-risk subgroup identification. Empirical evaluations on credit approval and fraud detection tasks demonstrate significant improvements in model validation accuracy and risk management efficacy.

This study addresses the challenge of effectively integrating radiologists’ assessments with AI predictions in mammographic screening to optimize rule-out and rule-in diagnostic strategies. It introduces, for the first time, a unified joint ROC theoretical framework tailored to both clinical scenarios. By modeling the dependence between physician and AI diagnostic outputs using bivariate copulas, the work theoretically derives—and empirically validates—the impact of their correlation on AUC performance: higher correlation improves rule-out efficacy in diseased populations, whereas lower correlation is preferable in non-diseased populations; conversely, for rule-in tasks, the opposite pattern holds. This framework provides a rigorous theoretical foundation and practical guidance for designing collaborative diagnostic systems that strategically leverage human–AI synergy.

This work investigates the geometric foundations of ROC and PR curves in binary classification, aiming to unify the understanding of curve morphology and classifier behavior through a geometric lens. Methodologically, it introduces the composite function (G = F_p circ F_n^{-1}) as a core modeling framework—where (F_p) and (F_n) denote the CDFs of positive and negative class score distributions—and rigorously establishes a geometric mapping between ROC/PR curve shapes and the underlying distributional geometry. It reveals that (G) quantifies inter-class leakage and admits interpretation via KL divergence. Furthermore, it derives geometric criteria for classifier dominance and interpretability grounded in differential geometry, statistical inference, and CDF transformation theory. The contributions include: (i) a principled, geometrically interpretable framework for threshold selection; (ii) robust, distribution-agnostic tools for classifier comparison; and (iii) enhanced reliability and adaptability in cost-sensitive deployment—particularly under class imbalance and distributional overlap.

This study addresses the overreliance on the Area Under the ROC Curve (AUC) in software defect prediction research, which can lead to biased model evaluation as AUC fails to capture a model’s discriminative performance across all classification thresholds. To overcome this limitation, the authors propose an augmented ROC curve and alternative visualization techniques that explicitly model the true positive rate and false positive rate as functions of the decision threshold, annotating corresponding threshold points directly on the ROC curve. Their analysis demonstrates that a high AUC does not guarantee superior performance over random guessing at every threshold, thereby exposing critical shortcomings of conventional evaluation practices. The work underscores the necessity of incorporating multi-threshold classification performance into a more comprehensive and nuanced assessment framework.

This study investigates how correlations among biomarkers influence the discriminative performance of predictive models, elucidating the mechanism by which adding new biomarkers does not necessarily improve model accuracy. Through theoretical derivations under multivariate normal and skewed distributions, simulation experiments—including log-folded bivariate normal and Gamma distributions—and validation using serum metabolomic data from pancreatic ductal adenocarcinoma patients, the work establishes, for the first time, an analytical relationship between biomarker correlation structures and the area under the ROC curve (AUC). The findings demonstrate that negative correlation most substantially enhances the joint AUC when individual biomarkers exhibit comparable predictive power, and real-world metabolomic data confirm that inter-biomarker correlation plays a decisive role in the performance of disease detection models.

This study addresses the lack of decision-oriented evaluation methodologies in current machine translation quality estimation (QE) systems. It introduces receiver operating characteristic (ROC) analysis into QE evaluation for the first time, complementing and validating against conventional metrics. Experimental results demonstrate that ROC analysis not only aligns consistently with existing evaluation outcomes but also yields actionable performance insights. By providing a clearer understanding of trade-offs between true positive and false positive rates across varying decision thresholds, this approach significantly enhances the practical utility of QE assessment and offers robust guidance for deployment decisions in real-world applications.

Conventional binary classification metrics (e.g., AUROC) suffer from representational bias under label-local exchange invariance (LxC), rendering them inadequate for LxC-invariant tasks such as causal discovery. Method: We propose LxCIM, the first evaluation metric explicitly designed for LxC invariance. We formally define LxC invariance and construct a unified framework satisfying ranking consistency, LxC invariance, computability, and multi-faceted interpretability. Leveraging insights from ROC and decision rate curves, we introduce the Cumulative Accuracy–Decision Rate Curve (CADRC) for fine-grained model behavior analysis. We rigorously establish theoretical connections between LxCIM and AUROC, accuracy, and AUDRC. Results: Experiments on bivariate causal discovery demonstrate that LxCIM significantly outperforms mainstream baselines, effectively mitigating evaluation bias. The implementation is open-sourced, confirming its robustness and practical utility.

Current evaluation practices for supervised learning models are often misleading due to an overreliance on single aggregate metrics, which neglect the alignment among data characteristics, task objectives, and real-world application contexts. This work reframes model evaluation as a context-dependent, decision-oriented process and systematically investigates—through controlled experiments—the impact of dataset properties, validation strategies, class imbalance, and asymmetric error costs on evaluation outcomes. Leveraging diverse benchmark datasets, multiple validation protocols, and multidimensional performance measures, the study uncovers common pitfalls such as the accuracy paradox, data leakage, and metric misuse. It proposes a structured evaluation framework explicitly aligned with operational goals, offering principled guidance for developing more robust, reliable, and trustworthy supervised learning systems.

This work addresses the long-standing challenge of model selection in anomaly detection under scenarios lacking labeled validation data, where existing approaches rely heavily on reconstruction quality or manual annotations. The authors propose AUCp, a fully unsupervised model selection metric that treats the entire unlabeled test set as positive (i.e., anomalous) and computes the Area Under the ROC Curve (AUC) accordingly. This approach enables, for the first time, completely annotation-free model selection across diverse unsupervised and self-supervised anomaly detection frameworks. Evaluated on medical image reconstruction tasks involving multiple neurological disorders and heterogeneous datasets, AUCp consistently outperforms conventional metrics, effectively identifying the best-performing inference models and thereby enhancing overall anomaly detection performance.

This study addresses the evaluation and selection of source-level likelihood ratio (LR) systems for forensic evidence-to-reference comparison tasks by proposing an integrated analytical framework that balances performance and practical feasibility. The authors employ strictly proper scoring rules to quantify how effectively each system updates Bayesian prior odds and present the first systematic comparison among specific-source feature-based, common-source anchored, and unanchored score-based LR approaches. Their findings reveal that specific-source feature-based LRs achieve the highest performance but incur substantial experimental costs, whereas common-source feature-based methods offer strong discriminative power with significantly reduced implementation complexity. All LR systems substantially outperform a baseline relying solely on prior odds. This work thus provides both theoretical grounding and practical guidance for selecting LR systems in forensic practice.