Mind the Generation Process: Fine-Grained Confidence Estimation During LLM Generation

2025-08-16

AI Summary

Large language models (LLMs) lack self-awareness and often exhibit overconfidence in erroneous outputs; existing confidence estimation methods are predominantly coarse-grained and discrete, hindering real-time reliability monitoring during generation. Method: We propose FineCE—a framework for fine-grained, continuous confidence estimation—incorporating a backward confidence integration (BCI) strategy and three optimal estimation position selection mechanisms. It jointly models token-level probability distributions, sequence-level scoring, and context-aware backtracking information via supervised learning. Contribution/Results: FineCE significantly outperforms state-of-the-art confidence estimation baselines across multiple benchmarks. It enhances the interpretability, controllability, and safety of LLM outputs by enabling precise, dynamic confidence assessment throughout the generation process—addressing critical gaps in trustworthy LLM deployment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

While large language models (LLMs) have demonstrated remarkable performance across diverse tasks, they fundamentally lack self-awareness and frequently exhibit overconfidence, assigning high confidence scores to incorrect predictions. Accurate confidence estimation is therefore critical for enhancing the trustworthiness and reliability of LLM-generated outputs. However, existing approaches suffer from coarse-grained scoring mechanisms that fail to provide fine-grained, continuous confidence estimates throughout the generation process. To address these limitations, we introduce FineCE, a novel confidence estimation method that delivers accurate, fine-grained confidence scores during text generation. Specifically, we first develop a comprehensive pipeline for constructing training data that effectively captures the underlying probabilistic distribution of LLM responses, and then train a model to predict confidence scores for arbitrary text sequences in a supervised manner. Furthermore, we propose a Backward Confidence Integration (BCI) strategy that leverages information from the subsequent text to enhance confidence estimation for the current sequence during inference. We also introduce three strategies for identifying optimal positions to perform confidence estimation within the generation process. Extensive experiments on multiple benchmark datasets demonstrate that FineCE consistently outperforms existing classical confidence estimation methods. Our code and all baselines used in the paper are available on GitHub.

Problem

• LLMs lack self-awareness and show overconfidence in predictions
• Existing confidence estimation methods are coarse-grained and discontinuous
• Need for fine-grained, continuous confidence scores during text generation

Innovation

• Fine-grained confidence estimation during LLM generation
• Backward Confidence Integration strategy for estimation
• Supervised training for predicting confidence scores