Speeding up Policy Simulation in Supply Chain RL

2024-06-04

arXiv.org

AI Summary

Strategy simulation in supply chain optimization suffers from prohibitively low efficiency—single monthly simulations require several hours. Method: This paper proposes a parallelization acceleration framework based on Picard iteration, the first application of such iteration to reinforcement learning (RL) policy simulation. Leveraging the contractive property of supply chain dynamics, the method ensures convergence within a fixed number of iterations independent of the time horizon, thereby decoupling long-horizon policy evaluation into independent, batch-parallelizable subtasks. The approach integrates GPU-accelerated batch processing, a distributed caching mechanism, and iterative state updates, accompanied by rigorous theoretical convergence guarantees. Contribution/Results: On large-scale supply chain optimization (SCO) benchmarks, the method achieves a 400× speedup on a single GPU. Its generalizability is validated across multiple RL environments, demonstrating consistent performance gains without sacrificing accuracy.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Simulating a single trajectory of a dynamical system under some state-dependent policy is a core bottleneck in policy optimization (PO) algorithms. The many inherently serial policy evaluations that must be performed in a single simulation constitute the bulk of this bottleneck. In applying PO to supply chain optimization (SCO) problems, simulating a single sample path corresponding to one month of a supply chain can take several hours. We present an iterative algorithm to accelerate policy simulation, dubbed Picard Iteration. This scheme carefully assigns policy evaluation tasks to independent processes. Within an iteration, any given process evaluates the policy only on its assigned tasks while assuming a certain"cached"evaluation for other tasks; the cache is updated at the end of the iteration. Implemented on GPUs, this scheme admits batched evaluation of the policy across a single trajectory. We prove that the structure afforded by many SCO problems allows convergence in a small number of iterations independent of the horizon. We demonstrate practical speedups of 400x on large-scale SCO problems even with a single GPU, and also demonstrate practical efficacy in other RL environments.

Problem

• Accelerates policy simulation in supply chains
• Reduces serial policy evaluation bottleneck
• Enables batched GPU-based policy evaluation

Innovation

• Picard Iteration algorithm
• GPU batched evaluation
• Independent policy task assignment