deep reinforcement learning

This paper addresses distributionally robust stochastic control in continuous state spaces to mitigate the policy fragility of conventional i.i.d. Markov models—arising from neglecting environmental input distribution shifts and endogenous dependencies. We propose a novel paradigm that balances modeling simplicity and robustness: adaptive adversarial perturbations are embedded within dynamic programming to unify *f*-divergence and Wasserstein-type ambiguity sets. We establish, for the first time in continuous spaces, a unified learning theory for robust value functions under both ambiguity-set classes, providing finite-sample minimax convergence rate bounds. Integrating distributionally robust optimization, stochastic control, and nonparametric statistics, we design a computationally tractable minimax policy learning algorithm. Experiments demonstrate that the framework achieves both statistical efficiency and strong robustness across real-world applications—including supply chain management and finance.

Classical dynamic programming fails for robust infinite-horizon MDPs under non-rectangular uncertainty sets, as it cannot accommodate their coupled, non-separable structure. Method: This paper introduces the first policy-gradient framework for such problems that simultaneously ensures global optimality and computational tractability. We propose a deterministic policy gradient method with provable error bounds quantifying deviation from rectangularity; design a robust Actor-Critic algorithm with controllable convergence rate; and introduce a novel metric characterizing the degree of non-rectangularity of uncertainty sets. Contribution/Results: We establish an $O(1/varepsilon^4)$ iteration complexity bound for convergence to an $varepsilon$-optimal policy, with rigorous global optimality guarantees. Numerical experiments on multiple benchmark tasks demonstrate that our approach significantly outperforms existing state-of-the-art methods.

To address low data efficiency and poor robustness in robot navigation policy learning under uncertainty, this paper proposes a Bayesian reinforcement learning framework integrating robust optimization with statistical robustness. Methodologically, it introduces (1) an unscented Bayesian optimization algorithm to ensure policy safety and reproducibility, and (2) a Boltzmann-based stochastic acquisition function coupled with an adaptive Gaussian process surrogate model to jointly enhance convergence and robustness against modeling errors. Evaluated on multiple benchmark functions and real-world legged locomotion tasks, the method achieves a 32% improvement in policy success rate, demonstrates significantly enhanced robustness to input perturbations and model mismatch, and substantially increases sample efficiency compared to existing approaches.

This paper addresses online robust reinforcement learning under dynamics mismatch between training and deployment environments, focusing on exploration challenges induced by dynamic uncertainty. We introduce the *supremal visitation ratio* to quantify discrepancies in environment dynamics and, within a distributionally robust MDP framework, propose the first efficient online algorithm achieving sublinear regret under an *f*-divergence ambiguity set—attaining optimal dependence in its regret bound. Theoretically, we establish matching upper and lower bounds on regret. Empirically, the algorithm demonstrates significant improvements over baseline methods across diverse dynamic shift scenarios, exhibiting both strong robustness and high sample efficiency.

Traditional reinforcement learning relies on state-dependent policies, struggling to simultaneously achieve theoretical interpretability and high-dimensional control performance. This work proposes a novel open-loop RL paradigm that abandons state feedback entirely and instead optimizes fixed-length action sequences directly, replacing step-wise decision-making with trajectory-level optimization. Methodologically, we establish a rigorous theoretical foundation grounded in Pontryagin’s Minimum Principle, circumventing the limitations of the Bellman equation. We design one robust model-based algorithm and two sample-efficient model-free algorithms, all accompanied by convergence guarantees. Empirical evaluation on the cart-pole swing-up task and two high-dimensional MuJoCo benchmarks demonstrates that our approach significantly outperforms mainstream baselines, validating its effectiveness, generalization capability, and theoretical consistency.

This work addresses the challenge of safe exploration in high-dimensional systems with unknown dynamics, where existing approaches offer only probabilistic safety guarantees in expectation and often fail to prevent safety violations in practice. To overcome this limitation, the authors propose a novel method that first learns a probabilistic control-affine dynamics model offline and explicitly incorporates model uncertainty into the design of Control Barrier Functions (CBFs), yielding conservative yet feasible safety constraints. During online execution, the CBF-based framework corrects policy actions to enforce hard safety guarantees without requiring prior knowledge of the true system dynamics. Empirical evaluations on multiple nonlinear continuous-control benchmarks demonstrate that the proposed approach significantly reduces safety violations while maintaining task performance comparable to state-of-the-art baselines.

To address low sample efficiency and significant sim-to-real gaps for complex robots operating in real-world environments, this paper proposes an online model-based reinforcement learning framework. The method dynamically constructs a dynamics model from real-time interaction data and integrates model predictive control with stochastic optimization for online policy adaptation. We provide theoretical guarantees showing a sublinear regret bound, ensuring continuous performance improvement and inherent adaptability to environmental dynamics. Experiments on a hydraulic excavator arm and a soft robotic manipulator demonstrate that the approach achieves performance comparable to state-of-the-art model-free methods within only a few hours of real-world interaction—substantially reducing sample requirements—while exhibiting strong robustness against abrupt disturbances such as payload changes. The core contribution is the first incorporation of rigorous online learning theory into a model-based RL architecture, uniquely balancing high sample efficiency, minimal reliance on simulation, and real-time adaptability.

To address robust decision-making in online reinforcement learning under distributional shifts, this paper proposes DR-RPO, a model-free policy optimization algorithm. DR-RPO is the first to integrate policy optimization with distributionally robust Markov decision process (MDP) theory, employing reference-policy regularization and a dual-constraint mechanism to jointly restrict both the policy space and transition dynamics, thereby enabling robust policy learning under adversarial environmental changes. Built upon the *d*-rectangular linear MDP framework, it combines linear function approximation, upper-confidence-bound (UCB) reward estimation, and policy gradients to support optimistic exploration in large state-action spaces. Theoretically, DR-RPO achieves sublinear regret and polynomial sample complexity. Empirically, it significantly outperforms existing methods across diverse distribution-shift scenarios, demonstrating both high robustness and superior sample efficiency.

This work addresses the challenge posed by time-varying system dynamics—such as those caused by wear or operating condition drift—that render conventional reinforcement learning methods ineffective. To tackle this issue, the paper proposes a model-based reinforcement learning approach tailored for non-stationary environments. The method integrates a Gaussian process dynamics model with an adaptive data buffering mechanism that explicitly limits the influence of outdated experiences, thereby enabling well-calibrated uncertainty estimation. Notably, it is the first to incorporate variable-budget non-stationarity analysis into a model-based control framework, offering dynamic regret guarantees. Empirical evaluations on multiple time-varying continuous control benchmarks demonstrate that the proposed algorithm significantly outperforms existing approaches, confirming its robustness and efficiency in non-stationary settings.

This work addresses the challenge of achieving robotic control that is both adaptive and robust in real-world environments characterized by environmental dynamics and epistemic uncertainty in reward signals. The authors propose the Distributionally Robust Free Energy Principle, which formalizes policy robustness under cognitive uncertainty and integrates it into a maximum entropy reinforcement learning framework, thereby unifying exploration and robustness. The approach requires no task-specific fine-tuning and enables zero-shot deployment. Combined with simulation-to-reality transfer techniques, the method achieves reproducible tabletop manipulation tasks on a Franka Emika robotic arm, substantially narrowing the sim-to-real gap and demonstrating strong effectiveness and generalization in continuous control settings.