How Much Imprecision is Enough Imprecision in my Classifier? A Practical Elicitation Procedure

2026-09-27

AI Summary

This study addresses the absence of operational tools for measuring user tolerance toward set-valued predictions in existing research. To this end, it proposes a user-oriented precision-robustness trade-off measurement mechanism that quantifies the degree to which users are willing to sacrifice predictive accuracy in exchange for robustness, thereby optimizing decision-making behavior. By integrating iterative parameter estimation with human-preference-inspired elicitation techniques, this mechanism pioneers a solution to fill the methodological gap in this domain. Experiments conducted on tabular and image benchmarks demonstrate that the proposed approach converges to target parameters with only a small number of samples, validating both its efficiency and practical applicability.

Abstract

Set-valued classifiers, whether derived from precise probabilities and an adapted cost function, from convex sets with a robust inference mechanism, or from conformal methods, are routine options to obtain more robust, trustworthy predictions. However, there is a lack of operational tools to measure how robust or imprecise a given user is ready to be when receiving predictions, that is how much precision he/she is ready to let go in exchange of more accuracy. This is why we propose, in this paper, practical and operational elicitation procedures to measure the user proneness to set-valued predictions. The effectiveness of the iterative elicitation procedure in converging to the target parameter value is demonstrated on both tabular and image datasets drawn from standard machine learning benchmarks. The results show that the procedure also presents the user with a small number of instances, highlighting the practicality of the approach for real-world applications aimed at identifying the decision maker's optimal behavior when faced with imprecision.

Problem

• set-valued classifiers
• imprecision
• elicitation procedure
• robustness
• user preference

Innovation

• Set-valued classifiers
• Imprecision elicitation
• Iterative procedure
• Robust predictions
• Decision making