Clustered Self-Assessment: A Simple yet Effective Method for Uncertainty Quantification in Large Language Models

2026-06-02

AI Summary

Large language models frequently generate factually incorrect responses and lack reliable uncertainty estimates, undermining users’ ability to assess output credibility. This work proposes a concise and efficient self-evaluation mechanism that quantifies uncertainty in an interpretable manner. By performing semantic clustering on a small number of sampled outputs, the method constructs structured multiple-choice questions and leverages the model’s probability distribution over the options to calibrate its confidence. Requiring only two additional samples, this approach consistently outperforms existing baselines across diverse models and datasets, achieving both high efficiency and superior performance in uncertainty estimation.

Abstract

Large language models (LLMs) demonstrate remarkable performance across diverse tasks, but they often generate responses that appear plausible while being factually incorrect. This problem is compounded by the lack of explicit uncertainty estimates, which makes it difficult for users to judge the reliability of model outputs. Existing uncertainty quantification methods typically rely on indirect signals, such as entropy across sampled generations. These signals can be difficult to interpret and do not fully leverage the model's ability to assess its own uncertainty. We propose a simple yet effective self-assessment method for uncertainty quantification in LLMs. Our approach groups sampled generations into semantically distinct clusters, converts them into answer options in a structured multiple-choice question, and uses the probability assigned by the LLM to each option as a confidence estimate. Experiments across multiple models and datasets show that our method consistently outperforms baseline approaches. Notably, it achieves competitive performance with as few as two additional samples, demonstrating both its effectiveness and efficiency.

Problem

• uncertainty quantification
• large language models
• self-assessment
• reliability
• confidence estimation

Innovation

• uncertainty quantification
• large language models
• self-assessment
• semantic clustering
• confidence estimation