🤖 AI Summary
This study addresses a critical yet overlooked issue in out-of-distribution (OOD) detection within evidential deep learning (EDL): the sensitivity of uncertainty-mass-based methods to the number of in-distribution (ID) classes. The authors systematically analyze how mismatches in cardinality between ID and OOD label sets introduce evaluation artifacts, particularly in causal language modeling and multiple-choice question answering (MCQA) tasks where ID/OOD boundaries are rigorously defined. They demonstrate for the first time that even a single-class difference can artificially inflate OOD detection performance, yielding AUROC and AUPR biases as high as 0.360 and 0.683, respectively. The work underscores the necessity of strictly controlling class cardinality to ensure reliable OOD evaluation in EDL frameworks and validates these findings across EDL, IB-EDL, and Dirichlet belief models on standard MCQA benchmarks.
📝 Abstract
Vacuity, or Uncertainty Mass (UM), is commonly used as a metric to evaluate Out-of-Distribution (OOD) detection in Evidential Deep Learning (EDL). It generally involves dividing the number of classes ($K$) by the total strength of belief ($S$) of the model's predictions, where $S$ is derived from summing the Dirichlet parameters. As such, UM is sensitive to the cardinality of $K$. In particular, it is unlikely in practice that there is a linear relationship between $K$ and $S$ as $K$ and $S$ increase due to the nature of EDL (suppressing incorrectly assigned evidence). As a result, when comparing In Distribution (ID) and OOD results, it is important that $K_{\mathrm{ID}}$ and $K_{\mathrm{OOD}}$ are equal; something that is not always ensured in practice. We provide an empirical demonstration of how results for AUROC and AUPR can substantially differ when class cardinality between ID and OOD differs by 1, with AUROC differing by as much as 0.318 and AUPR by 0.613 for standard EDL, and AUROC by 0.360 and AUPR by 0.683 for IB-EDL. More concretely, our findings isolate an evaluation artefact: when K differs between ID and OOD, AUROC/AUPR can be artificially inflated without any change in model predictions. We further discuss the evaluation of EDL over causal language models using Multiple-Choice Question-Answer (MCQA) datasets and argue for clearer definitions of ID and OOD in this context. Our primary contribution is an empirical and theoretical demonstration that vacuity-based OOD detection in EDL-fine-tuned LLMs is highly sensitive to uncontrolled differences in evaluated class cardinality.