mdp formulation

This paper addresses a class of multistage stochastic programming problems characterized by continuous state and action spaces, decision-dependent uncertainty, and limited statistical learning capability. To overcome the expressive limitations of conventional models, we propose an extended policy graph framework that explicitly captures the feedback effect of decisions on uncertainty and incorporates online learning mechanisms. Building upon this, we design a novel stochastic dual dynamic programming (SDDP) algorithm and its nonconvex approximation variant, tailored for efficiently solving such structured Markov decision processes. Experimental results on a suite of benchmark instances—increasing in complexity—demonstrate that our approach significantly improves policy quality and computational scalability. The work establishes a new paradigm for stochastic optimization that jointly integrates statistical learning with sequential decision-making, offering both enhanced expressiveness and tractability.

This work addresses the high computational complexity and inefficiency of value function approximation in high-dimensional structured Markov decision processes (MDPs). By revealing the low-dimensional geometric structure of decision tessellations induced by optimal policies, the authors propose a boundary-driven policy approximation method that directly learns policy regions rather than value functions. They further introduce a policy loss decomposition mechanism that quantitatively links performance degradation to action boundary errors. Evaluated on inventory control and queue admission tasks, the proposed approach significantly reduces policy error and value gap compared to standard reinforcement learning baselines, achieving faster error convergence and enhanced training stability.

This paper addresses non-cumulative Markov decision processes (NCMDPs), where the objective is to optimize the expectation of an arbitrary function—e.g., maximum reward, Sharpe ratio—of the reward sequence, rather than the conventional discounted cumulative reward. We propose the first general, theoretically rigorous state-augmentation mapping that equivalently transforms any NCMDP into a standard MDP. This reduction enables direct application of classical reinforcement learning algorithms (e.g., DQN, policy gradients) and dynamic programming methods. Empirical evaluation across diverse domains—including control, finance (portfolio optimization), and combinatorial optimization—demonstrates substantial improvements in final performance and training efficiency. Our core contribution is the establishment of a formal theoretical equivalence between NCMDPs and standard MDPs, accompanied by a scalable algorithmic framework for practical implementation. The approach unifies treatment of non-cumulative objectives within the standard RL paradigm while preserving computational tractability and theoretical soundness.

This work addresses non-stationary environments in real-world settings, where exogenous temporal events continuously induce dynamics—challenging the standard stationary MDP assumption in reinforcement learning. We formally introduce the *exogenously driven non-stationary MDP* (ED-NMDP), a novel framework that explicitly models environmental non-stationarity as governed by an exogenous temporal process. Within this framework, we propose a history-dependent policy conditioned on both agent states and the observed exogenous event sequence, and design a tailored policy iteration algorithm. Theoretically, we prove convergence of the algorithm under non-stationarity and derive a tight sample complexity upper bound that explicitly quantifies the impact of exogenous process intensity and memory decay. Empirically, our method demonstrates significantly improved policy robustness and adaptability on benchmark continuous-control tasks. Our core contribution is the establishment of a rigorous, analyzable, and verifiable theoretical foundation for exogenously driven non-stationary RL.

This work addresses the problem of learning non-stationary optimal policies for finite-horizon constrained Markov decision processes (CMDPs), filling a theoretical gap left by prior studies focused on infinite-horizon settings and stationary policies. We propose the first policy gradient algorithm specifically designed for finite-horizon CMDPs, employing time-varying parametric policy networks and Lagrangian relaxation to handle hard constraints. We provide rigorous convergence guarantees—proving that the algorithm converges to a constraint-optimal solution under standard regularity conditions. The method accommodates continuous state-action spaces, supports function approximation, and scales effectively to high-dimensional problems. Empirical evaluation across multiple benchmark tasks demonstrates substantial improvements in cumulative reward, constraint satisfaction rate, and convergence stability, thereby validating both the theoretical assurances and practical efficacy of the approach.

This work addresses the problem of policy synthesis in Markov decision processes (MDPs) under entropy-based constraints that enforce concentration of state visitation distributions. It formalizes entropy maximization as a policy synthesis objective for the first time, establishes its computational complexity, and introduces a novel method combining convex duality theory with invariant synthesis to handle nonlinear entropy constraints in a conditionally complete manner. By systematically analyzing the roles of memory and randomization in policies, the approach effectively synthesizes and verifies entropy-constrained policies across multiple benchmark instances, substantially extending the expressiveness and applicability of existing policy synthesis frameworks.

This work addresses the challenge of modeling long-term dependencies in non-Markovian decision processes, where observations and rewards depend on the full interaction history, rendering conventional reinforcement learning methods ineffective. The authors propose a reward-oriented Agent State Markov (ASM) policy framework that recursively updates an internal state representation and jointly optimizes this representation with the control policy. Within this framework, they establish the first policy gradient theorem applicable to non-Markovian environments. Building on this theoretical foundation, they develop an end-to-end ASMPG algorithm and provide rigorous proofs of its finite-time and almost sure convergence. Empirical evaluations demonstrate that the proposed method significantly outperforms baseline approaches relying on predictive objectives for state representation across multiple non-Markovian tasks, confirming the effectiveness and superiority of the ASM paradigm in handling history-dependent dynamics.

This work addresses the lack of a unified theoretical framework for reinforcement learning, which has hindered systematic analysis of its convergence, sample complexity, and generalization. Building upon Markov decision processes and Bellman operators, the paper introduces a cohesive analytical framework that integrates tools from operator theory, stochastic approximation, convex duality, and function approximation. This framework encompasses a broad range of algorithms, including value iteration, policy iteration, temporal difference methods, off-policy learning, and constrained MDPs. By leveraging contraction mappings, monotone operators, martingale techniques, mirror/proximal optimization, concentration inequalities, and mixing process theory, the study establishes finite-sample performance bounds and asymptotic convergence guarantees for diverse reinforcement learning algorithms, thereby forging a rigorous theoretical bridge between probability theory, optimization, and statistics.

This work addresses the challenge of learning near-optimal policies in partially observable Markov decision processes (POMDPs) using only finite observation-action histories. The authors propose a hyper-state MDP framework that enables efficient model estimation from a single trajectory and computes near-optimal finite-window policies via value iteration. A key theoretical contribution is the novel connection established between filter stability and concentration inequalities for weakly dependent random variables, which yields tight sample complexity guarantees for single-trajectory estimation in the hyper-state MDP. By integrating model-based reinforcement learning, hyper-state modeling, and analysis of non-independent sequences, the approach rigorously approximates high-performance policies in the original POMDP while maintaining strong theoretical foundations.

This work investigates how to achieve optimal policies in Markov decision processes (MDPs) that incorporate future information—such as reference trajectories or predictions—by leveraging model predictive control (MPC). The authors formulate MPC as a class of parameterized policies and train them end-to-end via reinforcement learning. Their key contribution lies in establishing, for the first time, the precise structural conditions under which MPC can exactly represent the optimal value function and policy, thereby providing a theoretical foundation for MPC as a structured function approximator with formal guarantees. Empirical validation on a point-mass racing task with future reference trajectories demonstrates that the proposed approach learns policies approaching optimality, confirming its effectiveness.