monte carlo methods

Addressing the challenges of expensive, stochastic, and analytically intractable function evaluations in reinforcement learning (RL) and approximate Bayesian computation (ABC), this paper systematically reviews and refactors the Monte Carlo methodology framework. We first unify surrogate modeling approaches—designed for costly, noisy, and intractable densities—into three principled categories, and propose a modular surrogate modeling paradigm that jointly optimizes accuracy, computational cost, and robustness. Our framework is innovatively extended to likelihood-free inference and online RL settings. Integrating Bayesian optimization, Gaussian processes, sequential Monte Carlo, importance sampling, and adaptive experimental design, we conduct comprehensive numerical experiments to quantitatively characterize the trade-offs among sample efficiency, convergence stability, and noise robustness. The results provide a reusable, principled guideline for method selection in RL policy evaluation and hyperparameter optimization.

This paper addresses the optimal design of proposal densities in Monte Carlo importance sampling, particularly under challenging settings involving dynamic updates and noise—such as in Approximate Bayesian Computation (ABC) and policy evaluation in reinforcement learning. It provides the first unified theoretical analysis of the applicability boundaries of multiple optimality criteria—including minimum variance and KL-divergence minimization—while establishing a cross-framework evaluation framework that jointly ensures theoretical guarantees and empirically comparable performance. Methodologically, the work integrates variational inference, sequential importance resampling, and annealed posterior modeling to propose a multi-proposal adaptive mechanism. Key contributions are: (1) necessary and sufficient conditions for proposal optimality across frameworks, with convergence guarantees; (2) systematic empirical validation of trade-offs among model selection accuracy, noise robustness, and computational efficiency in adaptive proposal design; and (3) an open-source empirical benchmark enabling reproducible, standardized comparison of future proposal mechanisms.

To address the low search efficiency and poor interpretability of strategies in complex combinatorial games, this paper proposes a multi-level simplification model-driven enhancement of Monte Carlo Tree Search (MCTS). Our method constructs progressive game simplification models and integrates heuristic performance prediction with policy ensembling to estimate algorithmic behavior within the simplified space, thereby guiding tree expansion and policy selection in the original MCTS. The core contribution is the first establishment of a closed-loop paradigm—“simplification modeling → performance prediction → search guidance”—unifying strategy predictability and interpretability. Evaluated on multiple challenging combinatorial game benchmarks, the proposed approach significantly improves MCTS’s strategic quality (average +12.7% win rate) and convergence speed (speedup ratio up to 2.3×), demonstrating the strong guiding efficacy of simplification-based analysis for solving complex games.

Monte Carlo Tree Search (MCTS) suffers from poor scalability on modern CPUs due to branch misprediction overhead and irregular memory access patterns induced by its pointer-based tree structure, limiting simulation throughput. To address this, we propose a compact array-based MCTS implementation that replaces the explicit pointer tree with an implicit array representation—preserving the UCT algorithm’s logic while eliminating indirect memory accesses and conditional branches. This design improves cache locality and instruction-level parallelism by enabling contiguous memory layouts and predictable control flow. Experimental evaluation demonstrates a substantial increase in simulations per unit time under identical clock cycles; maximum search depth improves by up to 2.8× compared to conventional pointer-based implementations. The approach significantly enhances MCTS scalability and real-time performance in latency-sensitive decision-making tasks, without compromising algorithmic correctness or generality.

Exact goodness-of-fit testing for discrete exponential family models suffers from computational intractability in high-dimensional sparse settings, primarily due to the difficulty of efficiently sampling lattice points uniformly from the constrained fiber. This paper introduces the first reinforcement learning framework for fiber sampling by formulating it as a Markov decision process. We propose an Actor-Critic–based sampling algorithm that integrates lattice geometry, exchangeable sampling mechanisms, and linear-algebraic acceleration techniques, accompanied by theoretical convergence guarantees. Compared with conventional MCMC and nonlinear algebraic approaches, our method substantially reduces computational overhead while generating statistically valid, exchangeable samples in high-dimensional sparse regimes. It thus overcomes a fundamental bottleneck in exact inference for large-scale structured data.

This paper addresses the NP-hard Weighted Vertex Coloring Problem (WVCP) by proposing a novel solution framework that integrates Monte Carlo Tree Search (MCTS) with multiple heuristic strategies. Departing from conventional random rollouts, the work systematically designs and evaluates MCTS variants employing greedy heuristics, local search, and combinatorial optimization rules during simulation. It presents the first systematic empirical study of heuristic-guided rollouts for WVCP, elucidating their mechanisms for improving solution quality and convergence speed, as well as identifying their practical applicability boundaries. Experimental results demonstrate that the best-performing heuristic-driven MCTS variant consistently outperforms classical heuristics and vanilla random MCTS on standard benchmarks. The framework offers an interpretable, efficient, and scalable paradigm for graph coloring and related combinatorial optimization problems.

Existing Hamiltonian Monte Carlo (HMC) methods for Bayesian models with implicit algebraic constraints embed numerical root-finding (e.g., Newton–Raphson) but suffer from high computational cost due to fixed initial guesses, requiring multiple solver iterations per HMC step. This work proposes a dynamic initial guess mechanism that adaptively updates the guess along the Hamiltonian trajectory. We introduce two heuristic strategies: solution-transferring extrapolation from previous iterates and gradient-guided extrapolation based on implicit differentiation. Leveraging JAX’s automatic differentiation and just-in-time compilation, we implement an efficient, differentiable numerical solver. Evaluated across diverse constrained models, our method accelerates NUTS sampling by 2–5×. We release *grapevine*, the first open-source library supporting constraint-aware HMC with dynamic initial guesses—establishing a scalable new paradigm for Bayesian inference in implicitly defined models.

This work addresses the inefficiency of traditional Monte Carlo Tree Search (MCTS) in adversarial tabletop games with high uncertainty—such as Jaipur, Lost Cities, and Splendor—where fixed determinization strategies hinder optimal use of computational resources. To overcome this limitation, the authors propose a dual-axis dynamic resource allocation mechanism integrated within determinized MCTS: it adaptively adjusts the number of determinization trees while allocating simulations non-uniformly based on knowledge gain, thereby enabling real-time optimization of the computational budget. Experimental results demonstrate that, under identical iteration and time constraints, the proposed approach yields statistically significant improvements in win rates, confirming its effectiveness and superiority in decision-making under uncertainty.

To address the low sampling efficiency of Markov Chain Monte Carlo (MCMC) methods caused by inherent sequential dependencies in long chains, this paper proposes the first general parallel-in-sequence MCMC framework. The core idea is to reformulate the state sequence as a fixed-point problem governed by a nonlinear recurrence, then solve it via parallel fixed-point iteration combined with quasi-Newton methods—thereby decoupling temporal ordering constraints across time steps. To this end, we design two memory-efficient parallel quasi-Newton algorithms, enabling parallelization of diverse samplers including Gibbs, Metropolis-Adjusted Langevin Algorithm (MALA), and Hamiltonian Monte Carlo (HMC). Experiments demonstrate speedups of tens to over one hundred times across multiple tasks: tens of parallel iterations suffice to generate hundreds of thousands of samples, achieving more than an order-of-magnitude improvement in overall performance and significantly overcoming the scalability bottleneck of conventional MCMC.

This work addresses the challenge of optimizing objectives with intractable gradients—such as maximum marginal likelihood estimation and generative model fine-tuning—by introducing, for the first time, a sequential Monte Carlo (SMC) sampler into a gradient-free optimization framework. The proposed approach replaces the conventional inner-loop sampling procedure to efficiently approximate stochastic gradients, substantially reducing computational overhead while preserving optimization performance. A theoretical convergence analysis is provided to support the method’s validity. Empirical evaluations across multiple energy-based models on reward tuning tasks demonstrate that the proposed algorithm achieves significant acceleration without compromising, and in some cases even improving, optimization quality.

In general-sum games with constrained computational resources or where deep reinforcement learning is inapplicable, existing Monte Carlo Tree Search (MCTS) variants—such as GRAVE—rely on manually tuned bias hyperparameters for exploration, limiting robustness and generalizability. Method: This paper proposes MCPS, an enhanced MCTS algorithm that replaces GRAVE’s parameter-sensitive bias term with permutation statistics over full-path actions in the exploration component. MCPS further integrates AMAF (All Moves As First) statistics, abstract action encoding, and a three-source weighted mechanism to jointly refine value estimation and balance exploration-exploitation. Results: Empirical evaluation demonstrates that MCPS significantly outperforms GRAVE in two-player games, achieves comparable performance in multi-player settings, and exhibits strong robustness to the critical ref hyperparameter—validating its generalization capability and practical utility across diverse game-theoretic domains.

In Monte Carlo Tree Search (MCTS), state/action abstraction often collapses multiple distinct actions into a single abstract node, leading to ambiguous action selection; conventional random tie-breaking yields suboptimal policies and degrades search efficiency. Method: We propose several novel intra-abstraction decision strategies that systematically refine action selection *within* abstract nodes, integrate them into a UCB-based MCTS framework, and synergistically combine them with imprecise abstraction techniques (e.g., pruned Optimistic Graph Abstraction). Contribution/Results: Extensive experiments across diverse benchmark environments and parameter configurations demonstrate that our strategies significantly outperform random tie-breaking baselines—accelerating convergence of abstraction-guided search, improving policy quality, and enhancing generalization stability. The approach establishes an interpretable, reusable internal decision paradigm for abstraction-augmented MCTS.