Exact Fast Batch Simulation for Tabular Reinforcement Learning

2026-10-03

AI Summary

This study addresses the computational redundancy and batch-size limitations inherent in traditional reinforcement learning simulation, which relies on independent trajectory generation for aggregated statistics. To overcome these challenges, this work proposes an exact and efficient batched simulation framework that replaces individual trajectory sampling with Markov flows. The core methodology integrates forward Markov flow sampling with a multivariate hypergeometric splitting algorithm, enabling adaptive simulation through recursive refinement. This approach reduces the computational complexity of finite-horizon tabular Markov decision processes from O(m) to O(1) or O(log m). Consequently, the proposed framework significantly accelerates simulator-based, offline, and online batched reinforcement learning, substantially lowering computational overhead while rigorously preserving statistical exactness.

Abstract

Simulation is a fundamental computational primitive in reinforcement learning (RL), yet conventional simulation explicitly generates individual trajectories even when downstream procedures use only aggregate statistics. To address this, we develop an exact fast-simulation framework for finite-horizon tabular Markov decision processes. Our framework has two complementary modes. In direct batch simulation, a batch is represented by its aggregate Markov flow. With sufficient parallel simulation resources, this flow can be obtained by trajectory aggregation; when such simulation is unavailable or costly but the initial state and transition distributions are directly accessible, we instead generate an identically distributed flow through forward Markov-flow sampling without materializing individual trajectories. The latter reduces the simulator-side computational dependence on batch size $m$ from $O(m)$ to $O(1)$. In adaptive batch simulation, when batch length is determined by a data-dependent condition, exact multivariate-hypergeometric splitting recursively refines a candidate Markov flow while preserving the conditional law, reducing the cost dependence on $m$ from $O(m)$ to $O(\log m)$. Together, these modes accelerate simulation by keeping trajectories aggregated whenever possible and refining flows only when required to locate data-dependent boundaries. The framework applies broadly across simulator-based, offline, and online batch or stage-based RL, as illustrated with representative algorithms from each setting.

Problem

• Reinforcement Learning
• Batch Simulation
• Tabular MDP
• Trajectory Aggregation
• Computational Efficiency

Innovation

• Tabular Reinforcement Learning
• Batch Simulation
• Markov Flow
• Multivariate-Hypergeometric Splitting
• Fast Simulation