Runtime Uncertainty Monitoring for LLM-Based Multi-Agent Systems Using Bayesian Networks

2026-07-28

AI Summary

This work addresses the uncertainty arising from model stochasticity and agent interdependence in large language model–driven multi-agent systems for actuarial risk modeling. To mitigate this, the authors propose a centralized hub-coordinated multi-agent framework, where specialized agents handle data preparation, modeling, review, and interpretation, respectively. The study introduces a novel approach that leverages length-normalized token-level log probabilities to generate calibrated task-level confidence scores, which are then integrated into a Bayesian network to enable runtime uncertainty propagation. While maintaining benchmark-level actuarial performance, the method effectively uncovers workflow stability characteristics and pathways of uncertainty propagation, thereby offering interpretable, real-time uncertainty monitoring capabilities crucial for high-stakes decision-making.

Abstract

This paper investigates how multi-agent systems (MAS)-based on large language models (LLMs) can support actuarial risk modelling, with a particular focus on uncertainty quantification. Actuarial workflows represent a high-stakes decision-support setting where unreliable outputs may lead to incorrect risk assessment, unfair pricing, and regulatory non-compliance. To address uncertainty introduced by the probabilistic nature of LLMs and dependencies between agents, a multi-agent framework is proposed in which specialised agents perform data preparation, modelling, review, and explanation tasks under a central hub. The main contribution is a novel approach to uncertainty propagation using token-level log-probabilities and a Bayesian Network. Importantly, log probabilities are not treated as direct probabilities of correctness or task success. Instead, length-normalised log-probability summaries are transformed into calibrated task-level confidence estimates before incorporation into the Bayesian Network. Results show that the framework reproduces baseline actuarial performance while providing additional insight into workflow stability and runtime uncertainty propagation.

Problem

• Runtime Uncertainty
• LLM-Based Multi-Agent Systems
• Uncertainty Quantification
• Actuarial Risk Modelling
• Bayesian Networks

Innovation

• Bayesian Networks
• Uncertainty Quantification
• Large Language Models
• Multi-Agent Systems
• Log-Probability Calibration