Extracting Probabilistic Knowledge from Large Language Models for Bayesian Network Parameterization

2025-05-21

AI Summary

Large language models (LLMs) implicitly encode event-level probabilistic knowledge, yet this knowledge remains systematically underutilized for probabilistic reasoning tasks such as Bayesian network (BN) parameterization. Method: We propose a novel LLM-based probabilistic knowledge extraction framework for automatic BN parameterization. It integrates (i) a conditionally guided prompting strategy leveraging next-token probability distributions, (ii) distributional calibration to mitigate LLM output bias, and (iii) a hybrid inference scheme that treats LLM outputs as expert priors, refined via few-shot empirical data. Contribution/Results: We introduce the first structured benchmark—comprising 80 publicly available BNs—for evaluating probabilistic knowledge extraction. Experiments demonstrate that our method significantly outperforms random, uniform, and raw token-probability baselines in low-data regimes, yielding more accurate and robust BN parameter estimates. This advances lightweight probabilistic modeling for high-stakes domains including healthcare and finance.

Technology Category

• Reasoning under Uncertainty: Relational Probabilistic Models
• Machine Learning: Probabilistic Circuits and Graphical Models
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models (LLMs) have demonstrated potential as factual knowledge bases; however, their capability to generate probabilistic knowledge about real-world events remains understudied. This paper investigates using probabilistic knowledge inherent in LLMs to derive probability estimates for statements concerning events and their interrelationships captured via a Bayesian Network (BN). Using LLMs in this context allows for the parameterization of BNs, enabling probabilistic modeling within specific domains. Experiments on eighty publicly available Bayesian Networks, from healthcare to finance, demonstrate that querying LLMs about the conditional probabilities of events provides meaningful results when compared to baselines, including random and uniform distributions, as well as approaches based on next-token generation probabilities. We explore how these LLM-derived distributions can serve as expert priors to refine distributions extracted from minimal data, significantly reducing systematic biases. Overall, this work introduces a promising strategy for automatically constructing Bayesian Networks by combining probabilistic knowledge extracted from LLMs with small amounts of real-world data. Additionally, we evaluate several prompting strategies for eliciting probabilistic knowledge from LLMs and establish the first comprehensive baseline for assessing LLM performance in extracting probabilistic knowledge.

Problem

• Extracting probabilistic knowledge from LLMs for Bayesian Networks
• Parameterizing BNs using LLM-derived probability estimates
• Combining LLM knowledge with minimal data to reduce biases

Innovation

• Extracting probabilistic knowledge from LLMs
• Parameterizing Bayesian Networks with LLMs
• Combining LLM knowledge with minimal real-world data