Technology Category
Intelligent Robots: Motion and Path PlanningPlanning, Routing, and Scheduling: Replanning and Plan RepairMultiagent Systems: Multiagent Planning
Hierarchical Deep Deterministic Policy Gradient for Autonomous Maze Navigation of Mobile Robots
🤖 AI Summary
To address the challenges of sparse rewards, inefficient exploration, and long-horizon planning in mobile robot maze navigation, this paper proposes a hierarchical Deep Deterministic Policy Gradient (DDPG) framework. The high-level policy generates semantic sub-goals, while the low-level policy executes fine-grained control actions. To mitigate sub-goal drift, historical experience relabeling is introduced; adaptive parameter-space noise and an integrated intrinsic–extrinsic reward shaping mechanism further enhance exploration robustness and policy stability. Building upon an improved DDPG architecture, the approach incorporates off-policy correction, gradient clipping, and Xavier initialization within a ROS/Gazebo simulation environment. Experimental evaluation across three distinct goal-reaching tasks demonstrates significant improvements: navigation success rate increases by ≥56.59%, and average cumulative reward rises by over 519.03—substantially outperforming standard DDPG and its prominent variants.
Technology Category
Application Category
📝 Abstract
Maze navigation is a fundamental challenge in robotics, requiring agents to traverse complex environments efficiently. While the Deep Deterministic Policy Gradient (DDPG) algorithm excels in control tasks, its performance in maze navigation suffers from sparse rewards, inefficient exploration, and long-horizon planning difficulties, often leading to low success rates and average rewards, sometimes even failing to achieve effective navigation. To address these limitations, this paper proposes an efficient Hierarchical DDPG (HDDPG) algorithm, which includes high-level and low-level policies. The high-level policy employs an advanced DDPG framework to generate intermediate subgoals from a long-term perspective and on a higher temporal scale. The low-level policy, also powered by the improved DDPG algorithm, generates primitive actions by observing current states and following the subgoal assigned by the high-level policy. The proposed method enhances stability with off-policy correction, refining subgoal assignments by relabeling historical experiences. Additionally, adaptive parameter space noise is utilized to improve exploration, and a reshaped intrinsic-extrinsic reward function is employed to boost learning efficiency. Further optimizations, including gradient clipping and Xavier initialization, are employed to improve robustness. The proposed algorithm is rigorously evaluated through numerical simulation experiments executed using the Robot Operating System (ROS) and Gazebo. Regarding the three distinct final targets in autonomous maze navigation tasks, HDDPG significantly overcomes the limitations of standard DDPG and its variants, improving the success rate by at least 56.59% and boosting the average reward by a minimum of 519.03 compared to baseline algorithms.
Problem
Research questions and friction points this paper is trying to address.
Enhances maze navigation with hierarchical DDPG for sparse rewards
Improves exploration and planning in complex robotic environments
Boosts success rates and rewards in autonomous maze tasks
Innovation
Methods, ideas, or system contributions that make the work stand out.
Hierarchical DDPG with high-low level policies
Off-policy correction enhances subgoal stability
Adaptive noise and reshaped rewards improve exploration
🔎 Similar Papers
2024-02-21arXiv.orgCitations: 0
