Exploring the Potential for Large Language Models to Demonstrate Rational Probabilistic Beliefs

2025-04-18

AI Summary

This work identifies systematic deficiencies in large language models’ (LLMs) probabilistic belief representation: their outputs frequently violate fundamental probability axioms—including the law of total probability and Bayesian updating—resulting in logically inconsistent and miscalibrated confidence estimates, thereby undermining their reliability in trustworthy decision-making and interpretable reasoning. To address this, the authors introduce the first benchmark dataset of statements with ground-truth uncertainty, accompanied by a multidimensional evaluation framework that assesses statement-level uncertainty annotation, confidence calibration, probabilistic logical consistency, and deviation from Bayesian updating. They conduct zero-shot and prompt-engineering evaluations across leading closed- and open-source LLMs. Experimental results demonstrate that current LLMs exhibit substantial deviations from rational probabilistic norms, with calibration errors significantly exceeding those of conventional statistical models—highlighting the urgent need for dedicated probabilistic cognitive modeling approaches.

Technology Category

• Reasoning under Uncertainty: Relational Probabilistic Models
• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Reasoning with Beliefs

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Advances in the general capabilities of large language models (LLMs) have led to their use for information retrieval, and as components in automated decision systems. A faithful representation of probabilistic reasoning in these models may be essential to ensure trustworthy, explainable and effective performance in these tasks. Despite previous work suggesting that LLMs can perform complex reasoning and well-calibrated uncertainty quantification, we find that current versions of this class of model lack the ability to provide rational and coherent representations of probabilistic beliefs. To demonstrate this, we introduce a novel dataset of claims with indeterminate truth values and apply a number of well-established techniques for uncertainty quantification to measure the ability of LLM's to adhere to fundamental properties of probabilistic reasoning.

Problem

• Assessing LLMs' ability for rational probabilistic beliefs
• Evaluating coherence in LLMs' uncertainty quantification
• Testing adherence to probabilistic reasoning principles

Innovation

• Novel dataset for indeterminate truth claims
• Established uncertainty quantification techniques
• Measuring LLM adherence to probabilistic reasoning