exploration strategies

This paper addresses the challenge of safe, optimal control for unknown dynamical systems under non-resettable, continuous online learning—where system resets are unavailable and safety must be guaranteed throughout operation. Method: We propose a non-episodic online learning framework leveraging a probabilistic dynamics model. It integrates pessimistic safety constraints with optimistic exploration to enable efficient learning while ensuring high-probability safety. Crucially, the method achieves finite-time, arbitrarily accurate learning of system dynamics, focusing exclusively on those dynamic features essential for optimal performance. Contributions/Results: Theoretical analysis guarantees strict safety satisfaction at all times and bounded, controllable learning error. Experiments on high-risk benchmarks—including autonomous racing cars and pneumatic-disturbance-affected UAV navigation—demonstrate rapid convergence under safety constraints and superior closed-loop control performance.

This paper addresses safe exploration for nonlinear robotic systems under unknown constraints. Method: We propose the first optimal control framework that jointly ensures safety and exploration completeness, integrating model predictive control (MPC), Lipschitz continuity analysis, goal-directed exploration, and receding-horizon replanning to enable online safety verification and proactive constraint avoidance. Contribution/Results: Theoretically, we establish the first finite-time sample complexity bound for general nonlinear systems—guaranteeing satisfaction of safety constraints with arbitrarily high probability and achieving exploration completeness within a finite number of samples. Empirically, we validate the framework in an autonomous driving simulation environment, demonstrating its safety, efficiency, and realizability of theoretical guarantees.

Deep reinforcement learning (DRL) achieves strong performance in control tasks, yet its exploratory training process frequently violates safety constraints, hindering real-world deployment; meanwhile, conventional safety-critical control methods rely on precise system dynamics models, which are often unavailable in practical scenarios. Method: We propose the first state-level safety policy optimization framework that requires no prior knowledge of system dynamics and guarantees zero safety violations throughout training. Our approach integrates a black-box safety monitor, a safety-set-guided exploration mechanism, and an imagined cost constraint into a differentiable policy gradient objective. Results: Experiments on high-dimensional robotic control tasks demonstrate strict adherence to state-level safety constraints, significantly outperforming existing safe DRL baselines while preserving efficient policy learning and robust safety assurance.

Reinforcement learning (RL) for robot navigation faces a fundamental trade-off between safety—particularly strict zero-collision guarantees—and task efficiency. Method: This paper proposes a Dynamic Safety Shield (DSS), wherein an RL agent adaptively tunes parameters online to tightly integrate model predictive control (MPC), constrained optimization, and deep RL (PPO/SAC), ensuring hard zero-collision constraints while jointly optimizing exploration efficiency and long-horizon task success. Contribution/Results: DSS achieves the first tight coupling of robust safety control with the RL policy’s long-horizon predictive capability, breaking the conventional safety–performance Pareto frontier. In simulation, it significantly outperforms state-of-the-art methods; real-robot experiments validate its practical efficacy. Compared to classical safety-shield approaches, DSS completes more navigation tasks; relative to constrained RL baselines, it reduces collision frequency by a substantial margin—all while maintaining strict safety guarantees.

In safe reinforcement learning, gradient conflicts between reward optimization and safety constraints hinder simultaneous achievement of high performance and strict safety compliance. To address this, we propose a gradient-manipulation-based soft-switching policy optimization method. Our approach is the first to systematically characterize the gradient conflict mechanism between reward and safety objectives, and to establish a theoretically grounded co-optimization framework with provable convergence guarantees. We further introduce Safety-MuJoCo, a novel benchmark for safe RL evaluation. The method integrates constrained policy optimization with gradient regularization, enabling performance improvement without compromising safety. Extensive experiments on Safety-MuJoCo and OmniSafe demonstrate that our method consistently outperforms state-of-the-art safe RL algorithms. Notably, it achieves Pareto-improved trade-offs: under high-reward policies, it maintains over 95% constraint satisfaction rates—significantly advancing the frontier of reward-safety balance.

This work addresses the challenge of safety constraint violations in long-horizon reinforcement learning tasks, which often arise from accumulated errors and limited exploration. To mitigate these issues, the paper proposes a novel safety-aware hierarchical reinforcement learning framework that integrates a learnable world model with a two-level policy architecture. The high-level policy generates safety-oriented subgoals, while the low-level policy leverages imagined rollouts within the learned predictive environment to evaluate and correct unsafe actions before execution, thereby enforcing safety at both levels. This approach is the first to incorporate imagination-based mechanisms into hierarchical reinforcement learning, effectively reducing error accumulation. Empirical results demonstrate that the method significantly improves constraint satisfaction rates and consistently adheres to predefined safety budgets in high-dimensional navigation and manipulation tasks, outperforming state-of-the-art safe reinforcement learning baselines.

This work addresses the challenge of catastrophic outcomes during exploration in reinforcement learning by proposing a safety-constrained Q-learning framework that strictly avoids visiting unsafe states. The approach enforces a safe set constraint on the behavior policy to guarantee that all trajectories remain within a predefined safe region throughout training. It introduces a two-stage training mechanism that separately optimizes the Q-function and the policy, along with a novel KL-regularized Bellman target that aligns the Q-function with the safe behavior policy. Additionally, a general parameterized policy extraction method is developed, applicable across diverse action spaces and policy classes. Empirical results demonstrate that the method completely eliminates unsafe state visits during training while achieving stable learning, accurate value estimation, and performance comparable to or better than existing baselines.

This work addresses the challenge of safe exploration in reinforcement learning by proposing a model-based safe reinforcement learning algorithm. The method approximates a worst-case optimization problem under uncertain dynamics by jointly imposing constraints over a finite set of dynamical samples and introduces a safety-aware exploration strategy based on epistemic uncertainty, eliminating the need for explicit exploration rewards. By integrating deep ensemble models to represent uncertainty, sampling-based model predictive control, and constrained optimization, the algorithm provides theoretical guarantees of high-probability safety throughout the entire learning process and establishes a finite-time sample complexity bound. Empirical results demonstrate that the approach achieves safe and efficient exploration on both simulated and real robotic platforms, successfully scales to high-dimensional continuous control tasks, and recovers near-optimal policies.

This work addresses the challenge of safe exploration in high-dimensional systems with unknown dynamics, where existing approaches offer only probabilistic safety guarantees in expectation and often fail to prevent safety violations in practice. To overcome this limitation, the authors propose a novel method that first learns a probabilistic control-affine dynamics model offline and explicitly incorporates model uncertainty into the design of Control Barrier Functions (CBFs), yielding conservative yet feasible safety constraints. During online execution, the CBF-based framework corrects policy actions to enforce hard safety guarantees without requiring prior knowledge of the true system dynamics. Empirical evaluations on multiple nonlinear continuous-control benchmarks demonstrate that the proposed approach significantly reduces safety violations while maintaining task performance comparable to state-of-the-art baselines.

This work addresses the challenge that reinforcement learning often fails to strictly satisfy hard safety constraints during exploration, risking irreversible damage to physical systems. To overcome this limitation, the authors propose a general safety framework that integrates deep reinforcement learning (DRL) with model predictive control (MPC). The approach leverages a known system dynamics model to construct an offline feasible set of state-action pairs and employs an online safety filter that projects the agent’s actions in real time onto this set, thereby providing formal safety guarantees throughout both training and deployment. By combining the safety assurances of MPC with the adaptive capabilities of DRL, the method enables safe exploration, stable policy convergence, and hardware protection, as demonstrated on a nonlinear single-degree-of-freedom experimental platform.