Partial VOROS: A Cost-aware Performance Metric for Binary Classifiers with Precision and Capacity Constraints

2025-10-21

AI Summary

Traditional ROC analysis and AUC ignore critical operational constraints in real-world deployment: minimum precision requirements (to mitigate alert fatigue), upper bounds on the number of positive predictions (due to limited human review capacity), and asymmetric costs of false positives versus false negatives. This paper proposes a resource-sensitive evaluation framework tailored to high-stakes settings such as clinical early-warning systems. First, it embeds precision and positive-prediction count constraints directly into ROC space to define the feasible classifier region. Second, it introduces VOROS—a monotonic, cost-sensitive metric that enables principled ranking across varying misclassification costs and constraint thresholds. Third, it constructs the feasible region via the ROC surface, integrating partial area accumulation with explicit cost modeling. Evaluated on mortality risk prediction using MIMIC-IV, the framework substantially outperforms conventional metrics, enabling more accurate identification of classifiers that satisfy clinical operational constraints and thereby enhancing practical deployability.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Metareasoning and Metaheuristics
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

The ROC curve is widely used to assess binary classification performance. Yet for some applications such as alert systems for hospitalized patient monitoring, conventional ROC analysis cannot capture crucial factors that impact deployment, such as enforcing a minimum precision constraint to avoid false alarm fatigue or imposing an upper bound on the number of predicted positives to represent the capacity of hospital staff. The usual area under the curve metric also does not reflect asymmetric costs for false positives and false negatives. In this paper we address all three of these issues. First, we show how the subset of classifiers that meet given precision and capacity constraints can be represented as a feasible region in ROC space. We establish the geometry of this feasible region. We then define the partial area of lesser classifiers, a performance metric that is monotonic with cost and only accounts for the feasible portion of ROC space. Averaging this area over a desired range of cost parameters results in the partial volume over the ROC surface, or partial VOROS. In experiments predicting mortality risk using vital sign history on the MIMIC-IV dataset, we show this cost-aware metric is better than alternatives for ranking classifiers in hospital alert applications.

Problem

• Addressing precision constraints to prevent false alarm fatigue
• Incorporating capacity limits for predicted positive classifications
• Accounting for asymmetric costs in false positives and negatives

Innovation

• Defines feasible ROC region with precision constraints
• Introduces partial area metric for cost-aware evaluation
• Proposes partial VOROS for classifier ranking under capacity limits