Problem
Research questions and friction points this paper is trying to address.
Deep Reinforcement Learning
High-Risk Motion Control
High-Precision Control
Irreversible Actions
Reward Landscape
Learning High-Risk High-Precision Motion Control
🤖 AI Summary
This study addresses the convergence difficulties of traditional reinforcement learning in high-risk motor control, where action irreversibility induces sharp reward landscapes. To overcome this challenge, we propose the SCOOT algorithm, which optimizes policies via advantage-weighted regression by integrating elite sample selection, a state-dependent mixture-of-experts (MoE) mechanism, and distance-regularized curriculum learning. Evaluated on a physics-based billiards task, our approach achieves exceptional action precision while autonomously discovering multiple effective striking strategies. By successfully navigating sparse, high-reward scenarios, this work establishes a novel paradigm for high-precision control in complex environments characterized by irreversible actions and rugged reward topologies.
📝 Abstract
Deep reinforcement learning (DRL) algorithms for movement control are typically evaluated and benchmarked on sequential decision tasks where imprecise actions may be corrected with later actions, thus allowing high returns with noisy actions. In contrast, we focus on an under-researched class of high-risk, high-precision motion control problems where actions carry irreversible outcomes, driving sharp peaks and ridges to plague the state-action reward landscape. Using computational pool as a representative example of such problems, we propose and evaluate State-Conditioned Shooting (SCOOT), a novel DRL algorithm that builds on advantage-weighted regression (AWR) with three key modifications: 1) Performing policy optimization only using elite samples, allowing the policy to better latch on to the rare high-reward action samples; 2) Utilizing a mixture-of-experts (MoE) policy, to allow switching between reward landscape modes depending on the state; 3) Adding a distance regularization term and a learning curriculum to encourage exploring diverse strategies before adapting to the most advantageous samples. We showcase our features’ performance in learning physically-based billiard shots demonstrating high action precision and discovering multiple shot strategies for a given ball configuration.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Deep Reinforcement Learning
High-Risk High-Precision Control
Advantage-Weighted Regression
Mixture-of-Experts
Computational Pool
🔎 Similar Papers
No similar papers found.
