Utility-Directed Conformal Prediction: A Decision-Aware Framework for Actionable Uncertainty Quantification

2024-10-02

AI Summary

Existing uncertainty quantification methods neglect downstream decision costs, leading to suboptimal decisions in high-risk scenarios. This paper proposes a utility-oriented conformal prediction framework that, for the first time, rigorously incorporates user-specified downstream utility functions into conformal prediction—jointly ensuring the standard 1−α statistical coverage guarantee and optimizing decision cost. To preserve semantic coherence of prediction sets, we introduce hierarchical label modeling (e.g., dermatological taxonomies) and design a model-agnostic permutation test coupled with a cost-sensitive set-shrinking algorithm. Empirical evaluation across multiple datasets demonstrates an average 23.6% reduction in decision cost. In a real-world dermatological diagnosis task, our method produces clinically interpretable and triage-appropriate diagnostic sets. The core contribution lies in achieving a rigorous unification of statistical reliability and decision utility—bridging theoretical guarantees with practical operational impact.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Decision/Utility Theory
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Interest has been growing in decision-focused machine learning methods which train models to account for how their predictions are used in downstream optimization problems. Doing so can often improve performance on subsequent decision problems. However, current methods for uncertainty quantification do not incorporate any information about downstream decisions. We develop a methodology based on conformal prediction to identify prediction sets that account for a downstream cost function, making them more appropriate to inform high-stakes decision-making. Our approach harnesses the strengths of conformal methods -- modularity, model-agnosticism, and statistical coverage guarantees -- while incorporating downstream decisions and user-specified utility functions. We prove that our methods retain standard coverage guarantees. Empirical evaluation across a range of datasets and utility metrics demonstrates that our methods achieve significantly lower costs than standard conformal methods. We present a real-world use case in healthcare diagnosis, where our method effectively incorporates the hierarchical structure of dermatological diseases. The method successfully generates sets with coherent diagnostic meaning, potentially aiding triage for dermatology diagnosis and illustrating how our method can ground high-stakes decision-making employing domain knowledge.

Problem

• Incorporates downstream decisions into uncertainty quantification.
• Improves decision-making with utility-directed conformal prediction.
• Applies method to healthcare for coherent diagnostic sets.

Innovation

• Conformal prediction for decision-aware uncertainty quantification
• Incorporates downstream cost functions in prediction sets
• Modular, model-agnostic with statistical coverage guarantees