From Trajectories to Instructions: Language-Conditioned Meta-Reinforcement Learning

2026-07-21

AI Summary

This work addresses the high computational cost and inefficiency of traditional meta-reinforcement learning, which relies on trajectory collection and gradient updates in the inner loop. The authors propose LA-MAML, a novel approach that leverages task-specific language instructions as direct signals within the inner loop, enabling rapid task adaptation without requiring environmental interaction or gradient-based updates. By embedding language instructions to modulate policy network parameters in a single step, LA-MAML deeply integrates linguistic guidance with policy learning within the MAML framework. Evaluated on the BabyAI benchmark, the method achieves performance comparable to or better than existing baselines while substantially reducing per-iteration training time, thereby significantly enhancing meta-training efficiency.

Abstract

Model-Agnostic Meta-Learning (MAML) is a widely used framework for reinforcement learning (RL) that enables efficient transfer by learning global policy parameters that can be rapidly adapted to new tasks. MAML training proceeds in two loops: an inner loop where the global parameters are adapted to task-specific parameters, and an outer loop where these task-specific parameters are evaluated and losses are back-propagated to improve the global parameters. Traditionally, the inner loop adaptation is performed by collecting trajectories from the task environment and applying gradient updates on the empirical expected return, which can be a costly operation. We note that it is the outer loop that drives the actual learning of global parameters, and therefore the inner loop adaptation mechanism need not be restricted to be gradient-based. This observation leads us to ask: Can we replace the inner loop trajectory collection and gradient update with a simpler, task-specific signal? In many practical settings, tasks are naturally accompanied by language instructions. Leveraging these instructions as a direct task-specific signal, we propose LA-MAML (Language Adapted MAML), which modifies the inner loop by adapting the global policy parameters in a single step through a learned embedding of the task instruction, replacing the inner loop trajectory collection and gradient-based updates. Experiments on the BabyAI benchmark demonstrate that LA-MAML achieves competitive or improved performance compared to baselines at a significantly lower per-iteration wall-clock training time. These results demonstrate that language instructions are an effective and efficient substitute for trajectory-based inner loop adaptation in meta RL.

Problem

• meta-reinforcement learning
• MAML
• inner loop adaptation
• language instructions
• trajectory collection

Innovation

• Language-Conditioned RL
• Meta-Reinforcement Learning
• MAML
• Instruction Embedding
• Inner Loop Acceleration