🤖 AI Summary
This study addresses the challenge of effectively evaluating the overall decision quality of autonomous agents in dynamic environments. Building upon the partially observable Markov decision process (POMDP) framework, the authors decompose decision-making into five components—information, belief, prediction, action, and utility—and systematically extend model risk management principles to autonomous AI systems. They propose a novel taxonomy encompassing six categories of risk related to state space representation, filtering, prediction, policy, and other critical aspects, while formally characterizing large language models as approximate Bayesian filters. Empirical validation in portfolio management, integrating the Black-Litterman model, belief calibration, coverage tests, and sensitivity analyses, demonstrates that accurate latent state inference significantly enhances decision performance. The results remain robust across a wide range of parameters, underscoring the framework’s effectiveness for AI governance and monitoring.
📝 Abstract
Agentic artificial intelligence systems introduce a new class of model risk. Unlike traditional predictive models, autonomous agents continuously acquire information, form beliefs regarding latent states of the environment, generate forecasts, select actions, and adapt their behavior over time. Existing validation methodologies focus primarily on predictive accuracy and therefore provide limited insight into the quality of the underlying decision process. This paper proposes a model validation framework for agentic AI based on Partially Observable Markov Decision Processes (POMDPs). The framework decomposes autonomous decision making into information, beliefs, forecasts, actions, and utility, allowing each component to be validated independently. Large language models (LLMs) are formalized as approximate Bayesian filtering operators, and a model-risk taxonomy is developed encompassing state-space, filtering, forecast, policy, utility-specification, and parameter risks.
The model risk validation methodology is demonstrated through a portfolio-management case study in which an agent infers latent market regimes from market and macroeconomic information, generates belief-conditioned forecasts, and constructs portfolios using a Black--Litterman framework. Empirical validation combines performance analysis, belief calibration diagnostics, coverage tests, ablation studies, and parameter-sensitivity analysis. The results indicate that latent-state inference contributes independently to decision quality and that the principal conclusions remain robust across a broad range of parameter values. The principal contribution of the paper is a practical framework for extending established model risk management concepts to autonomous AI systems and providing a rigorous foundation for their validation, governance, and monitoring.