LEACL: LLM-Enhanced Automatic Curriculum Learning for Reinforcement Learning in Long-Horizon Manipulation Tasks

2026-07-26

AI Summary

This work addresses the challenges of sparse rewards and high task complexity in long-horizon manipulation tasks, where conventional automated curriculum learning (ACL) methods suffer from limited generalization due to reliance on handcrafted task parameters and difficulty metrics. The authors propose a novel approach that integrates large language models (LLMs) with ACL: the LLM automatically decomposes complex tasks into subtasks and generates corresponding formal specifications, enabling ACL to guide agent learning using only sparse rewards. This method is the first to leverage an LLM for both task decomposition and subtask specification generation, eliminating the need for manually engineered dense reward functions. Evaluated on five long-horizon tasks from the LIBERO benchmark, the approach significantly outperforms baselines that use hand-designed dense rewards, demonstrating superior automation and generalization capabilities.

Abstract

Long-horizon manipulation tasks pose significant challenges for reinforcement learning due to sparse reward signals and long horizons. Automatic curriculum learning (ACL) has been proposed to tackle these challenges by progressively training agents on a sequence of tasks, from easier to more difficult. However, the success of ACL depends heavily on task-dependent specifications-such as well-defined task parameter spaces and difficulty measures-which are often manually crafted and difficult to generalize across diverse tasks. Recent advances in large language models (LLMs) offer a promising alternative by enabling the decomposition of complex tasks into meaningful subtasks using the LLMs' web-scale common-sense knowledge. This decomposition can provide a natural curriculum structure for efficient learning of long-horizon tasks. However, existing LLM-based methods typically rely on hand-designed dense reward functions to learn each subtask, which can introduce bias and still requires significant human supervision. In this work, we propose LLM-enhanced automatic curriculum learning (LEACL), a framework that integrates LLMs and ACL to address these limitations. Specifically, LLMs are used to both decompose tasks into subtasks and to generate task-dependent specifications for each subtask. These specifications are then used by ACL algorithms to guide learning using only sparse reward signals, eliminating the need for dense reward design. We evaluate LEACL on five long-horizon manipulation tasks from the LIBERO benchmark. LEACL achieves better asymptotic performance in terms of the success rates compared to human-designed dense rewards.

Problem

• long-horizon manipulation tasks
• sparse reward signals
• automatic curriculum learning
• task decomposition
• dense reward design

Innovation

• LLM-enhanced curriculum learning
• automatic curriculum learning
• sparse reward reinforcement learning
• long-horizon manipulation
• task decomposition