🤖 AI Summary
This work investigates how self-supervised objectives can enhance an agent’s exploration and adaptation capabilities in out-of-distribution tasks during unsupervised pre-training. To this end, the authors propose ULEE, a method that integrates in-context reinforcement learning, adversarial goal generation, and performance-estimation-driven dynamic curriculum scheduling within a meta-learning framework to enable frontier-aware adaptive curriculum learning. Experiments on the XLand-MiniGrid benchmark demonstrate that ULEE substantially outperforms training from scratch, DIAYN, and other curriculum-based approaches, exhibiting superior zero-shot and few-shot generalization across novel goals, environmental dynamics, and map structures. Furthermore, ULEE provides high-quality initial policies that facilitate effective downstream fine-tuning.
📝 Abstract
Unsupervised pre-training can equip reinforcement learning agents with prior knowledge and accelerate learning in downstream tasks. A promising direction, grounded in human development, investigates agents that learn by setting and pursuing their own goals. The core challenge lies in how to effectively generate, select, and learn from such goals. Our focus is on broad distributions of downstream tasks where solving every task zero-shot is infeasible. Such settings naturally arise when the target tasks lie outside of the pre-training distribution or when their identities are unknown to the agent. In this work, we (i) optimize for efficient multi-episode exploration and adaptation within a meta-learning framework, and (ii) guide the training curriculum with evolving estimates of the agent's post-adaptation performance. We present ULEE, an unsupervised meta-learning method that combines an in-context learner with an adversarial goal-generation strategy that maintains training at the frontier of the agent's capabilities. On XLand-MiniGrid benchmarks, ULEE pre-training yields improved exploration and adaptation abilities that generalize to novel objectives, environment dynamics, and map structures. The resulting policy attains improved zero-shot and few-shot performance, and provides a strong initialization for longer fine-tuning processes. It outperforms learning from scratch, DIAYN pre-training, and alternative curricula.