🤖 AI Summary
This paper addresses the problem of selecting appropriate belief state approximations for sample-based planning and calibration in latent-state simulators accessible only via sampling. Methodologically, it reformulates belief selection as a conditional distribution selection problem, uncovering a fundamental theoretical distinction between observation and latent-state spaces regarding identifiability and learnability. It precisely characterizes the applicability boundaries of single-reset versus repeated-reset strategies and proposes a theoretically grounded distribution selection algorithm that explicitly accounts for distribution shift and coupling with rollout policies. Contributions include: (i) a rigorous analysis of how distribution selection affects rollout performance; (ii) identification of a sharp success/failure dichotomy between two conditional distribution selection paradigms under different reset mechanisms; and (iii) novel principled criteria for trustworthy planning in latent-state simulators. The results establish formal guarantees on approximation quality and planning reliability under sampling constraints.
📝 Abstract
State resetting is a fundamental but often overlooked capability of simulators. It supports sample-based planning by allowing resets to previously encountered simulation states, and enables calibration of simulators using real data by resetting to states observed in real-system traces. While often taken for granted, state resetting in complex simulators can be nontrivial: when the simulator comes with latent variables (states), state resetting requires sampling from the posterior over the latent state given the observable history, a.k.a. the belief state (Silver and Veness, 2010). While exact sampling is often infeasible, many approximate belief-state samplers can be constructed, raising the question of how to select among them using only sampling access to the simulator.
In this paper, we show that this problem reduces to a general conditional distribution-selection task and develop a new algorithm and analysis under sampling-only access. Building on this reduction, the belief-state selection problem admits two different formulations: latent state-based selection, which directly targets the conditional distribution of the latent state, and observation-based selection, which targets the induced distribution over the observation. Interestingly, these formulations differ in how their guarantees interact with the downstream roll-out methods: perhaps surprisingly, observation-based selection may fail under the most natural roll-out method (which we call Single-Reset) but enjoys guarantees under the less conventional alternative (which we call Repeated-Reset). Together with discussion on issues such as distribution shift and the choice of sampling policies, our paper reveals a rich landscape of algorithmic choices, theoretical nuances, and open questions, in this seemingly simple problem.