Random Latent Exploration for Deep Reinforcement Learning

2024-07-18

International Conference on Machine Learning

AI Summary

This work addresses insufficient exploration in deep reinforcement learning under sparse-reward settings. We propose a reward-free exploration method guided by stochastic latent-space goals. Our approach uniquely integrates random latent goal sampling with reward-free policy shaping—achieving intrinsic-motivation-like deep exploration without introducing any auxiliary intrinsic reward module. By jointly modeling the latent space and learning goal-conditioned policies, we construct an end-to-end, plug-and-play exploration component. Evaluated on both discrete-action Atari and continuous-control Isaac Gym benchmarks, our method consistently outperforms standard noise-injection baselines (e.g., parameter or action space noise) and reward-shaping approaches in average performance. Crucially, it achieves this while maintaining architectural simplicity and strong generalization across diverse environments and tasks.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We introduce Random Latent Exploration (RLE), a simple yet effective exploration strategy in reinforcement learning (RL). On average, RLE outperforms noise-based methods, which perturb the agent's actions, and bonus-based exploration, which rewards the agent for attempting novel behaviors. The core idea of RLE is to encourage the agent to explore different parts of the environment by pursuing randomly sampled goals in a latent space. RLE is as simple as noise-based methods, as it avoids complex bonus calculations but retains the deep exploration benefits of bonus-based methods. Our experiments show that RLE improves performance on average in both discrete (e.g., Atari) and continuous control tasks (e.g., Isaac Gym), enhancing exploration while remaining a simple and general plug-in for existing RL algorithms.

Problem

• Enhance exploration in reinforcement learning
• Simplify exploration strategy complexity
• Improve performance in control tasks

Innovation

• Random Latent Exploration strategy
• Simplifies exploration in RL
• Enhances performance in control tasks