Bridging Deep Reinforcement Learning and Motion Planning for Model-Free Navigation in Cluttered Environments

2025-04-09

AI Summary

To address insufficient exploration in deep reinforcement learning (DRL) navigation under complex, cluttered environments—caused by sparse rewards and system disturbances—this paper proposes a graph-guided dense reward framework. It integrates generic graph-structured motion planning into model-free DRL (PPO/SAC) for the first time, constructing a fully covered dense graph via state-space discretization and generating globally dense reward signals through graph traversal. This significantly mitigates exploration failure under sparse-reward conditions. Evaluated across multiple high-clutter simulated environments, the method improves task success rate by 42% and sample efficiency by 3.1×, while enhancing policy robustness against disturbances. The core innovation lies in formulating structured graph planning as a differentiable reward generator, enabling a synergistic closed-loop between exploration guidance and policy optimization.

Technology Category

• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Machine Learning: Graph-based Machine Learning
• Intelligent Robots: Motion and Path Planning

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Deep Reinforcement Learning (DRL) has emerged as a powerful model-free paradigm for learning optimal policies. However, in real-world navigation tasks, DRL methods often suffer from insufficient exploration, particularly in cluttered environments with sparse rewards or complex dynamics under system disturbances. To address this challenge, we bridge general graph-based motion planning with DRL, enabling agents to explore cluttered spaces more effectively and achieve desired navigation performance. Specifically, we design a dense reward function grounded in a graph structure that spans the entire state space. This graph provides rich guidance, steering the agent toward optimal strategies. We validate our approach in challenging environments, demonstrating substantial improvements in exploration efficiency and task success rates. The project website is available at: https://plen1lune.github.io/overcome_exploration/

Problem

• Model-free navigation in cluttered environments using DRL
• Insufficient exploration in sparse-reward or dynamic environments
• Bridging motion planning with DRL for better guidance

Innovation

• Combines graph-based motion planning with DRL
• Designs dense reward function using graph structure
• Improves exploration efficiency in cluttered environments