Conditional Deep Generative Models for Belief State Planning

2025-05-16

AI Summary

For high-dimensional continuous-state partially observable Markov decision processes (POMDPs), conventional particle filtering struggles to accurately represent posterior beliefs. This work introduces conditional deep generative models (cDGMs) for POMDP belief representation—the first such application—enabling flexible modeling of complex posterior distributions over high-dimensional continuous state-observation spaces and scalable sample generation. Our method is trained end-to-end on stochastic rollout trajectory data, learning the mapping from observation sequences to belief distributions in a data-driven manner. Evaluated on a large-scale continuous mineral exploration POMDP, the proposed approach significantly outperforms particle filtering baselines in both belief estimation accuracy and downstream planning performance. It effectively alleviates the expressivity bottleneck in belief representation and inference for high-dimensional POMDPs.

Technology Category

• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)
• Reasoning under Uncertainty: Relational Probabilistic Models
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Partially observable Markov decision processes (POMDPs) are used to model a wide range of applications, including robotics, autonomous vehicles, and subsurface problems. However, accurately representing the belief is difficult for POMDPs with high-dimensional states. In this paper, we propose a novel approach that uses conditional deep generative models (cDGMs) to represent the belief. Unlike traditional belief representations, cDGMs are well-suited for high-dimensional states and large numbers of observations, and they can generate an arbitrary number of samples from the posterior belief. We train the cDGMs on data produced by random rollout trajectories and show their effectiveness in solving a mineral exploration POMDP with a large and continuous state space. The cDGMs outperform particle filter baselines in both task-agnostic measures of belief accuracy as well as in planning performance.

Problem

• Representing high-dimensional belief states in POMDPs
• Handling large continuous state spaces in POMDPs
• Improving belief accuracy and planning performance

Innovation

• Uses conditional deep generative models (cDGMs)
• Trains cDGMs on random rollout trajectories
• Outperforms particle filter baselines in accuracy