Weakly Time-Coupled Approximation of Markov Decision Processes

2026-03-13

AI Summary

This work addresses the intractable computational complexity of high-dimensional finite-horizon Markov decision processes (MDPs), which escalates severely with increasing time horizons. To overcome this limitation, the authors propose a Weakly Time-Coupled Approximation (WTCA) architecture that decouples inter-stage dependencies, thereby rendering computational complexity independent of horizon length. The approach reveals that temporal coupling in existing approximation schemes stems from structural design choices and constructs a WTCA with a provably tighter upper bound. Theoretical analysis shows that WTCA outperforms Approximate Linear Programming (ALP), closely matches Path Optimization (PO), and converges to optimality as the basis function set expands. Integrating parallel block coordinate descent with Least-Squares Monte Carlo under stochastic environments, WTCA yields tighter upper bounds than both PO and Least-Squares Monte Carlo (LSM) in Bermuda option pricing and ethanol production case studies, while generating near-optimal policies over long horizons.

Technology Category

• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)
• Reasoning under Uncertainty: Stochastic Optimization
• Multiagent Systems: Mechanism Design

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Finite-horizon Markov decision processes (MDPs) with high-dimensional exogenous uncertainty and endogenous states arise in operations and finance, including the valuation and exercise of Bermudan and real options, but face a scalability barrier as computational complexity grows with the horizon. A common approximation represents the value function using basis functions, but methods for fitting weights treat cross-stage optimization differently. Least squares Monte Carlo (LSM) fits weights via backward recursion and regression, avoiding joint optimization but accumulating error over the horizon. Approximate linear programming (ALP) and pathwise optimization (PO) jointly fit weights to produce upper bounds, but temporal coupling causes computational complexity to grow with the horizon. We show this coupling is an artifact of the approximation architecture, and develop a weakly time-coupled approximation (WTCA) where cross-stage dependence is independent of horizon. For any fixed basis function set, the WTCA upper bound is tighter than that of ALP and looser than that of PO, and converges to the optimal policy value as the basis family expands. We extend parallel deterministic block coordinate descent to the stochastic MDP setting exploiting weak temporal coupling. Applied to WTCA, weak coupling yields computational complexity independent of the horizon. Within equal time budget, solving WTCA accommodates more exogenous samples or basis functions than PO, yielding tighter bounds despite PO being tighter for fixed samples and basis functions. On Bermudan option and ethanol production instances, WTCA produces tighter upper bounds than PO and LSM in every instance tested, with near-optimal policies at longer horizons.

Problem

• Markov decision processes
• finite-horizon
• high-dimensional uncertainty
• computational scalability
• temporal coupling

Innovation

• Weakly Time-Coupled Approximation
• Markov Decision Processes
• Scalability
• Upper Bounds
• Stochastic Optimization