Problem
Research questions and friction points this paper is trying to address.
sparse reward
robot policy learning
reward shaping
physical interaction
Learning Robot Policies from Sparse Success Signals via STL-Guided Stein Variational Policy Gradient
🤖 AI Summary
This study addresses the challenges in robotic policy learning where local progress under sparse rewards rarely translates into task completion and complex physical interactions are difficult to coordinate. To overcome these issues, this work proposes STL-SVPG, a method that innovatively integrates Signal Temporal Logic (STL) into Stein Variational Policy Gradient (SVPG). Specifically, it replaces local reward shaping with smoothed STL robustness as a trajectory-level objective and leverages model-based dynamics backpropagation to achieve precise global credit assignment. Evaluated across six UAV and manipulator benchmarks, the proposed approach attains the highest success rates on five task categories. Furthermore, the policies learned in simulation successfully transfer to real-world deployment for executing complex temporal and contact-rich tasks.
📝 Abstract
Learning robot policies for tasks with sparse success signals is challenging when completion depends on coordinated actions, precise contact outcomes, or satisfying several conditions together. Intricate physical interactions with the world further complicate these requirements. Prior work using conventional reward shaping mechanisms provides dense feedback but local progress might not translate into eventual task completion. We present Signal Temporal Logic-guided Stein Variational Policy Gradient (STL-SVPG), a population-based method that uses smooth STL robustness as a trajectory-level training objective. Differentiating this objective through the dynamics assigns credit to policy actions according to their effect on the complete task specification, rather than local progress alone. We evaluate the approach on six quadcopter and manipulator tasks that involves event-triggered responses, strictly ordered behavior, responses within specified deadlines, and physical interaction with the world. STL-SVPG achieves the highest mean success rate among the compared methods on five of six benchmarks. Simulation-trained policies trained in simulation transfer temporal and contact task behavior to the real world.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Signal Temporal Logic
Stein Variational Policy Gradient
Sparse Reward
Credit Assignment
Sim-to-Real Transfer
🔎 Similar Papers
No similar papers found.
