Adaptive Reward Design for Reinforcement Learning in Complex Robotic Tasks

2024-12-14

arXiv.org

AI Summary

In reinforcement learning, Linear Temporal Logic (LTL) tasks suffer from sparse rewards, hindering subgoal guidance, slowing policy convergence, and degrading robustness. To address this, we propose a progress-aware adaptive reward shaping method: for the first time, we quantify LTL satisfaction progress as a continuous, differentiable reward signal and design an online mechanism to dynamically update the reward function according to the agent’s current learning state. Our approach integrates LTL task compilation, formal progress modeling, and deep RL frameworks (PPO/SAC). Evaluated across multiple benchmark environments, the method significantly accelerates convergence, increases average expected return by 23%, improves task completion rate by 31%, and outperforms both traditional sparse-reward baselines and handcrafted reward-shaping approaches in terms of robustness and generalization.

Technology Category

• Machine Learning: Reinforcement Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Planning, Routing, and Scheduling: Learning for Planning and Scheduling

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

There is a surge of interest in using formal languages such as Linear Temporal Logic (LTL) to precisely and succinctly specify complex tasks and derive reward functions for Reinforcement Learning (RL). However, existing methods often assign sparse rewards (e.g., giving a reward of 1 only if a task is completed and 0 otherwise). By providing feedback solely upon task completion, these methods fail to encourage successful subtask completion. This is particularly problematic in environments with inherent uncertainty, where task completion may be unreliable despite progress on intermediate goals. To address this limitation, we propose a suite of reward functions that incentivize an RL agent to complete a task specified by an LTL formula as much as possible, and develop an adaptive reward shaping approach that dynamically updates reward functions during the learning process. Experimental results on a range of benchmark RL environments demonstrate that the proposed approach generally outperforms baselines, achieving earlier convergence to a better policy with higher expected return and task completion rate.

Problem

• Sparse rewards in RL fail to encourage subtask completion
• LTL-specified tasks need adaptive reward shaping for better performance
• Dynamic reward updates improve convergence and task completion rates

Innovation

• Uses LTL for precise task specification
• Develops adaptive reward shaping approach
• Dynamically updates reward functions