Innovation
Methods, ideas, or system contributions that make the work stand out.
Credal Width
Random-Set Language Models
Feature Attribution
Expected Gradients
Epistemic Uncertainty
XU-RS: Explaining Credal Width in Random-Set Language Models
🤖 AI Summary
This study addresses the lack of interpretability in language model uncertainty and the difficulty of determining whether such uncertainty relies on relevant input features by proposing the XU-RS framework. This method achieves, for the first time, fine-grained attribution of credal width—a measure of epistemic uncertainty derived from random sets—for pretrained language models. Specifically, it employs expected gradients to quantify uncertainty at the input token level and reveals how normalization influences answer sets. Experiments on the MedQA dataset, incorporating a zero-masking technique, demonstrate that masking high-ranking tokens significantly alters the credal width, outperforming random baselines. These results validate the effectiveness of the proposed attribution approach in interpreting the sources of model uncertainty.
📝 Abstract
Uncertainty estimates tell us how unsure a model is, but not why. Without knowing which parts of an input influences a model's uncertainty, we cannot tell whether that uncertainty score depends on input features that are relevant for the task. We study this problem in randomset classifiers built using pretrained language models. These classifiers assign probability to individual answers and to groups of answers, producing lower and upper probabilities for each answer; The difference between these probabilities, called credal width, is used to represent epistemic uncertainty about an answer arising from limited training data. We propose XU-RS, a framework that attributes an answer's credal width to the input tokens (words or word pieces) supplied to a language model. XU-RS uses Expected Gradients (a standard feature attribution method) to estimate how input tokens contribute to credal width. The proposed framework is evaluated on a MedQA dataset using SmolLM3-3B and Llama-2-7B models, demonstrating that setting the embedding of a token ranked highly by XU-RS to zero (zero-masking) causes larger changes in credal width than zero-masking randomly selected tokens. In addition, we show that normalisation can cause other answer groups to influence an answer's width, reveal how token attribution can mask numerical errors, and provide diagnostic checks to verify whether a token ranked highly by XU-RS meaningfully explains model uncertainty.
🔎 Similar Papers
No similar papers found.
