Graph-Based Stochastic Power-UCT: Monte-Carlo Graph Search with Power Mean Estimation

2026-09-17

AI Summary

本文针对树搜索中状态重复导致的样本浪费问题，提出了一种基于图的随机Power-UCT方法，通过共享相同深度的状态来提高样本效率。

Abstract

Tree-based Monte-Carlo Tree Search (MCTS) duplicates the same state when it is reached through different trajectories, which can waste simulations in stochastic MDPs. We introduce Graph-Based Stochastic-Power-UCT (GS-Power-UCT), which shares states reached at the same planning depth while keeping separate values for states reached at different depths. This design applies to general stochastic MDPs, including problems with cycles. We prove that for a fixed planning horizon, the root estimate converges to the finite-horizon value at rate $O(n^{-1/2})$, matching tree-based Stochastic-Power-UCT while reusing samples across shared states. We also study two full-state variants: GS-Power-UCT-F, which stores one node per physical state to increase sample sharing but may mix values from different remaining horizons, and GS-Power-UCT-F$^+$, which uses an adaptive horizon to control this bias. The latter converges to $V^{\star}(s_0)$, the optimal infinite-horizon discounted value at the root state $s_0$, when the remaining cross-depth gap vanishes. Experiments on stochastic planning benchmarks show improved sample efficiency over tree-based and graph-based baselines.

Problem

• Monte-Carlo Tree Search
• Stochastic MDPs
• Sample Efficiency

Innovation

• Graph-Based Stochastic-Power-UCT
• Sample Reuse
• Stochastic MDPs
• Adaptive Horizon